23  December  1999 


MILITARY  OPERATIONS 
RESEARCH  SOCIETY 


MILITARY  OPERATIONS  RESEARCH  SOCIETY 


r 


Joint  Experimentation 

Mini-Symposium  and  Workshop 
8-11  March  1999 
AFSC,  Norfolk,  VA 

Chair 

Dr.  David  S.  Alberts 


UNCLASSIFIED 
Approved  for  Public  Release 


101  South  Whiting  Street  ♦  Suite  202  ♦  Alexandria  Virginia  22304-3416 
(703)751-7290  ♦  FAX:  (703)751-8171  ♦  email:  morsofflce@aol.com 
URL:  http://www-mors.org 


20020830  079 


DISCLAIMER 


This  Military  Operations  Research  Society  proceeding  report  summarizes  the 
findings  of  a  Mini-Symposium  conducted  over  four  days  by  experts,  users  and 
participants  interested  in  joint  experimentation.  It  is  not  intended  to  be  a 
comprehensive  treatise  on  the  subject.  It  reflects  the  major  concerns,  insights, 
thoughts  and  directions  of  the  participants  at  the  time  of  the  mini-symposium. 


CAVEATS 

•  The  Military  Operations  Research  Society  does  not  make  nor  advocate 
official  policy. 

•  Matters  discussed  or  statements  made  during  the  Mini-Symposium 
were  the  sole  responsibility  of  the  participants  involved. 

•  The  Society  retains  all  rights  regarding  final  decisions  on  the  content 
of  this  mini-symposium  proceeding 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMBNo.  0704-0188 

Public  reporting  burden  for  tills  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any 
other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations 
and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  budget.  Paperwork  Reduction  Project  (0704-0188), 
Washington,  DC  20503 

1 .  AGENCY  USE  ONLY  (Leave  blank) 

2.  REPORT  DATE 

23  December  1999 

3.  REPORT  TYPE  AND  DATES  COVERED 

Mini-Symposium  and  Workshop  Proceedings 

8-11  March  1999 

4.  TITLE  AND  SOBITTLE 

Joint  Experimentation  Mini-Symposium  and  Workshop  Proceedings 

5.  FUNDING  NUMBERS 

O&MN 

6.  AUTHOR(S) 

Corrina  Ross  Witkowski  and  Jarvey  Nelson,  Editors 

Corrina  Ross  Witkowski,  Publisher 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Military  Operations  Research  Society,  Inc. 

101  S.  Whiting  Street,  Suite  202 

Alexandria  VA  22304-3416 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSORING/MONirORING 
AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Distribution  Statement  A: 

Approved  for  Public  Release;  Distribution  Unlimited 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  words) 

This  publication  is  the  proceedings  from  the  Joint  Experimentation  Mini-Symposium  and 
Workshop. 

14.  SUBJECT  TERMS 

15.  NUMBER  OF  PAGES 

i-vi,  1-134,  A-1  to  C-14 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 

OF  REPORT 

UNCLASSIFIED 

18.  SECURITY  CLASSIFICATION 

OF  THIS  PAGE 

UNCLASSIFIED 

19.  SECURITY  CLASSIFICATION  OF 
ABSTRACT 

UNCLASSIFIED 

20.  UMITATION  OF  ABSTRACT 

UNLIMITED 

NSN  7540-01-280-5500  Standard  Form  298  (Rev.  2-89) 


Prescribed  by  ANSI  Sta.  239-18 
298-102 


SECURriY  CLASSIFICATION  OF  THIS  PAGE 


CLASSIFIED  BY: 


DECLASSIFIED  ON: 


MILITARY 
OPERATIONS 
RESEARCH  SOCIETY 


m 

MILITARY  OPERATIONS  RESEARCH  SOCIETY 


Joint  Experimentation 

Mini-Symposium  and  Workshop 
8-11  March  1999 
AFSC,  Norfolk,  VA 

Chair 

Dr.  David  S.  Alberts 


UNCLASSIFIED 
Approved  for  Public  Release 


101  South  Whiting  Street  ♦  Suite  202  ♦  Alexandria  Virginia  22304-3416 
(703)751-7290  ♦  FAX:  (703)751-8171  ♦  email:  morsofrice@aol.com 
URL:  http://wwvv.mors.org 


11 


The  Military  Operations  Research  Society  (MORS) 


The  purpose  of  the  Military  Operations  Research  Society  (MORS)  is  to  enhance  the 
quality  and  effectiveness  of  classified  and  unclassified  military  operations  research.  To 
accomplish  this  purpose,  the  Society  provides  media  for  professional  exchange  and  peer 
criticism  among  students,  theoreticians,  practitioners  and  users  of  military  operations 
research.  These  media  consist  primarily  of  the  traditional  annual  MORS  Symposia 
(classified),  their  published  abstracts  or  proceedings,  special  mini-symposia,  workshops, 
colloquia  and  special  purpose  monographs  and  other  publications.  MORS  publishes  two 
quarterly  periodicals,  PHALANX  and  Military  Operations  Research.  PHALANX  is  the 
MORS  bulletin  and  Military  Operations  Research  is  a  referred  journal.  The  forum 
provided  by  these  media  is  directed  to  display  the  state  of  the  art,  to  encourage  consistent 
professional  quality,  to  stimulate  communication  and  interaction  between  practitioners 
and  users,  and  to  foster  the  interest  and  development  of  students  of  operations  research. 
In  performing  its  function,  the  Military  Operations  Research  Society  does  not  make  or 
advocate  official  policy  nor  does  it  attempt  to  influence  the  formulation  of  policy. 
Matters  discussed  or  statements  made  during  the  course  of  its  symposia  or  printed  in  its 
publications  represent  the  positions  of  the  individual  participants  and  authors  and  not  of 
the  Society. 

The  Military  Operations  Research  Society  is  operated  by  a  Board  of  Directors  consisting 
of  30  members,  28  of  whom  are  elected  by  vote  of  the  Board  to  serve  a  term  of  four 
years.  The  persons  nominated  for  this  election  are  normally  individuals  who  have 
attained  recognition  and  prominence  in  the  field  of  military  operations  research  and  who 
have  demonstrated  an  active  interest  in  its  programs  and  activities.  The  remaining  two 
members  of  the  Board  of  Directors  are  the  Immediate  Past  President  who  serves  by  right 
and  the  Executive  Vice  President  who  serves  as  a  consequence  of  his  position.  A  limited 
number  of  Advisory  Directors  are  appointed  from  time  to  time,  usually  for  a  one-year 
term,  to  perform  some  particular  function. 

MORS  is  Sponsored  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director,  Assessment  Division,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director  of  Command  and  Control,  Deputy  Chief  of  Staff,  Air  and  Space 
Operations,  US  Air  Force 

•  The  Commanding  General,  Marine  Corps  Combat  Development  Command 

•  The  Director  of  Force  Structure,  Resources  and  Assessment,  The  Joint  Staff 

•  The  Director  Program  Analysis  and  Evaluation,  Office  Secretary  of  Defense 


III 


iv 


Table  of  Contents 


Military  Operations  Research  Society . iii 

Executive  Summary . 1 

Background . 1 

Mini-Symposium  Synopsis . 1 

Workshop  Organization . 1 

Major  Findings . 2 

Nature  of  Joint  Experimentation . 2 

Turning  Lessons  Recorded  into  Lessons  Learned . 2 

Leveraging  Experimental  Events . 3 

Organize  for  Success . 3 

Need  for  Balanced  Experimentation . 3 

Education  and  Training . 3 

Planning . 4 

Preparation  and  Conduct . 4 

Design  of  Experiments . 5 

Modeling  and  Simulation . 5 

Assessment  and  Utilization  of  Results . 6 

The  Way  Ahead . 6 

Working  Group  Reports 

Design  of  Experiments . 7 

Planning  Experiments . 21 

Conduct  of  Experiments . 37 

Modeling  and  Simulation . 63 

Assessment  and  Utilization  of  Results . 77 

Report  of  the  Synthesis  Working  Group . 115 

Appendix  A:  Acronyms . A-1 

Appendix  B;  Terms  of  Reference . B-1 

Appendix  C:  Participants . . . C-1 


V 


vi 


Executive  Summary 


Background 

The  DoD  has  embarked  upon  an 
ambitious  program  of  experimentation  to 
support  the  transformation  of  the 
military  to  meet  the  Century  mission 
challenges.  There  is  limited  expertise 
and  experience  in  designing,  conducting 
and  exploiting  the  results  of  large-scale 
experiments.  The  analytical  community 
has  much  to  offer  but  needs  to  adapt  its 
methods,  approaches  and  tools  in  order 
to  support  this  program  of  Joint 
Experimentation.  The  goal  of  this 
special  MORS  meeting,  attended  by 
approximately  200  people,  was  to 
contribute  to  planning,  conduct  and 
exploitation  of  Joint  Experimentation  by 
leveraging  the  experience  and  expertise 
of  the  analytical  community.  The  mini¬ 
symposium  was  designed  to  educate  the 
DoD  analytical  community  about  the 
nature  of  military  experimentation, 
recently  completed  events,  experimental 
plans  and  the  issues  involved  and  future 
challenges.  The  workshop  was  designed 
to  identify  critical  issues  and  develop 
conclusions  and  recommendations  to 
improve  Joint  Experimentation.  A 
unique  aspect  was  the  full  participation 
of  the  US  Atlantic  Command  (ACOM), 
the  DoD’s  executive  agent  for  Joint 
Experimentation.  ACOM’s  participation 
helped  ensure  that  the  workshop  focused 
upon  issues  of  concern  —  and  is 
expected  to  facilitate  the  adoption  of  the 
workshop’s  recommendations. 

Mini-Symposium  Synopsis 

Given  the  diversity  of  the  participants, 
the  significant  number  of  ongoing 
activities  related  to  Service 


experimentation  and  the  quickly  moving 
program  in  Joint  Experimentation,  it  was 
felt  that  the  workshop  needed  to  be 
preceded  by  a  mini-symposium  to  bring 
participants  up  to  speed  and  achieve  a 
common  level  of  understanding  of  the 
issues.  Presentations  by  LTG  Keane, 
DCINC,  ACOM,  and  Maj  Gen  Peppe, 

J9,  ACOM,  acquainted  participants  with 
an  overview  of  the  goals  and  objectives 
of  Joint  Experimentation  and  current 
plans.  Mr.  Eash,  DUSD  (Advanced 
Concepts  and  Systems),  provided 
information  about  the  ACTD  program. 
Mr.  Starry,  EDA  JAWP,  discussed  the 
generation  of  concepts  to  be  explored  by 
Joint  Experimentation.  Dr.  Alberts, 
(OASD(C3I)),  discussed  the  nature  of 
OR  and  challenges  of  Joint 
Experimentation.  Three  Service  panels 
were  convened.  The  first  addressed 
Service  Experimentation,  the  second. 
Service  Laboratories  and  related 
activities,  and  the  third  addressed  how 
the  Services  were  structuring  to  work 
with  ACOM  on  Joint  Experimentation. 

Workshop  Organization 

Workshop  participants  were  organized 
into  six  working  groups.  A  working 
group  was  assigned  to  each  of  the  phases 
of  experimentation:  plaiming,  design, 
preparation  and  conduct  and 
analysis/exploitation  of  results.  In 
addition,  a  working  group  looked  at  how 
modeling  and  simulation  could 
contribute  to  each  of  these  phases  of 
experimentation.  There  was  also  a 
synthesis  working  group  that  looked  for 
common  findings  across  these  working 
groups  and  provided  an  independent 
assessment  of  the  major  issues. 
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Major  Findings 

Nature  of  Joint  Experimentation 

The  scale  and  complexity  of  Joint 
Experimentation  currently  envisioned 
presents  its  own  set  of  challenges.  These 
challenges  are  compounded  by 
widespread  cultural  and  organizational 
implications.  With  the  goal  of  creating, 
nurturing  and  maturing  innovative 
operational  concepts  and  applications  of 
technology,  Joint  Experimentation  will, 
in  order  to  be  successful,  require  an 
unprecedented  change  to  the  status  quo. 
This  report  will  focus  on  only  one 
dimension  of  change,  the  one  that  affects 
the  way  we  plan,  design,  conduct  and 
utilize  the  results  of  “experimental” 
campaigns  and  events.  Change  is 
required  because  the  purpose  and  nature 
of  experimentation  is  different  from  the 
purposes  and  natures  of  exercises,  tests, 
evaluations  and  demonstrations  that  we 
are  so  used  to  conducting. 

Joint  Experimentation  is  an  iterative 
process  of  collecting,  developing, 
exploring  and  maturing  concepts  to 
identify,  demonstrate  and  recommend 
value-added  solutions  for  changes  to 
DOTMLP  (Doctrine,  Organization, 
Training,  Materiel,  Leadership,  and 
Personnel)  to  achieve  significant 
advances  in  future  joint  operational 
capabilities.  Discovery  is  an  integral 
part  of  this  process,  as  is  the  testing  of 
hypotheses  and  the  use  of 
experimentation  to  demonstrate  a 
capability.  Given  these  different  aspects 
of  experimentation,  the  various 
community  “cultures”  brought  to  the 
Joint  Experimentation  table  need  to 
adapt  new  values,  ideas  and  methods. 
Notions  of  what  constitutes  success  and 
failure,  the  role  of  the  assessor,  the 


tradeoffs  between  free-play  and  control 
all  need  to  be  examined  in  a  new  light. 

Central  to  the  effective  synthesis  of  on¬ 
going  Service  and  Joint  Experimentation 
into  a  coherent  body  of  knowledge  is  the 
ability  of  all  in  the  community  to 
communicate  effectively  with  one 
another.  Differences  in  culture  and 
language  make  this  quite  difficult.  A 
common  understanding  of  the  nature  of 
experimentation,  the  development  of  a 
new  “experimental  culture”  and  a 
universal  lexicon  of  terms  would  go  a 
long  way  to  addressing  these  problems. 

Turning  Lessons  Recorded  into 
Lessons  Learned 

There  is  a  growing  body  of  “lessons 
recorded”  coming  out  of  Service 
experimentation.  These  lessons  cover 
every  phase  and  aspect  of 
experimentation  including  the  time  it 
takes  to  plan,  the  process  for  screening 
experimental  hypotheses  and  issues 
associated  with  conduct  and  assessment. 
Several  “show  stoppers”  have  been 
identified  which  alone  can  significantly 
degrade  the  utility  of  an  experiment. 

Two  key  examples  are:  1)  failure  to  train 
the  participants  to  an  appropriate  level  of 
proficiency  on  either  the  systems  or  the 
concepts,  or  2)  failure  to  ensure  that  the 
experimental  inffastmcture  (the  systems) 
is  properly  functioning.  Either  of  these 
failures  can  prevent  the  needed  data 
collection,  thus  making  the  experimental 
results  meaningless.  In  addition,  many 
relatively  small  things  can  go  wrong 
with  an  experiment.  While  any  single 
one  may  not  derail  the  train,  the 
combination  of  many  “duck  bites”  can 
seriously  degrade  the  value  of  the 
experiment.  Having  at  least  a 
preliminary  Code  of  Best  Practice 
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(COBP)  and  access  to  lessons  recorded 
during  similar  events  could  make  a  big 
difference  in  the  outcome. 

Leveraging  Experimental  Events 

Particularly  in  the  next  few  years,  the 
Joint  Experimentation  Plan  relies  heavily 
on  its  ability  to  leverage  a  large  number 
of  Service  events.  Given  resource  and 
schedule  constraints,  the  concept  of 
leveraging  is  very  attractive  in  theory. 
However,  based  on  the  military 
commimity’s  experience,  it  is  very 
difficult  in  practice  and  the  results  are 
usually  disappointing.  To  improve  the 
chances  of  success,  a  number  of 
important  prerequisites  must  be  satisfied. 
They  include  high-level  agreements  on 
objectives  and  priorities,  written  MOUs, 
integrated  planning  and  conduct, 
deconfliction  of  scenario  events  and 
controllables  and  adequate  resources  and 
training. 

Organize  for  Success 

Given  the  iterative  nature  of 
experimentation  and  the  complex 
interactions  among  design,  conduct  and 
assessment  activities,  it  is  critical  to  have 
a  single  integrated  team  stay  with  an 
experiment  fi'om  concept  to  completion. 
The  same  is  true  for  the  development 
and  maturation  of  a  concept  that 
involves  synthesizing  the  results  from 
many  experimental  events.  It  is 
recommended  that  the  ACOM  J9 
consider  reorganizing  to  accomplish  this. 

Need  for  Balanced  Experimentation 

It  has  been  noted  that  many  experiments 
have  not  achieved  a  proper  balance 
among  pre-experiment  activities,  the 
conduct  of  the  experiment  and  post¬ 


experiment  activities.  As  a  result  of 
insufficient  resources  being  applied  to 
the  pre-  and  post-  phases  of  experiments, 
experiments  have  not  been  as  well 
focused,  instrumented  and/or  executed 
as  they  could  have  been,  and  the  data 
collected  was  not  analyzed  as  thoroughly 
as  it  could  have  been.  The  bottom  line  is 
that  there  was  a  reduced  ability  to  draw 
conclusions  or  reduced  confidence  in  the 
conclusions  that  were  drawn.  Both  pre- 
and  post-experiment  activities  can 
contribute  significantly  to  success  by 
focusing  on  the  interesting  parts  of 
scenario  space  and  by  performing  “what 
if’  and  sensitivity  analyses  to  leverage 
the  limited  observations  obtained  during 
the  “live”  part  of  the  experiment.  This  is 
an  extension  of  the  model-test-model 
paradigm  developed  by  the  OT&E. 

Balance  is  also  needed  across  the 
functional  components  of 
experimentation.  Insufficient  resources 
and  time  given  to  training  and 
assessment,  for  example,  can  seriously 
impair  an  experiment’s  ability  to  provide 
useful  results. 

Education  and  Training 

Almost  without  exception  there  has  not 
been  adequate  education  and  training  for 
participants.  All  participants  need  to 
understand  what  the  experiment  is  all 
about,  what  to  expect,  and  how  to  be 
proficient  in  their  assigned  roles.  This 
includes  the  users  of  experimental 
results,  the  director  of  the  experiment, 
the  players,  controllers  and  assessors. 
Lack  of  education  and  training  can  be  a 
real  “show  stopper.”  It  is  recommended 
that  education  and  training  be  conducted 
and  that  levels  of  understanding  and 
proficiency  be  assessed  as  adequate 


before  an  experiment  is  allowed  to 
proceed. 

Planning 

To  ensure  that  experiments  are 
successful,  adequate  attention  needs  to 
be  paid  to  each  of  the  following:  concept 
development,  concept  exploration  and 
refinement  and  the  development  of  an 
experimental  strategy.  “Concept 
development”  involves  defining  a 
statement  of  an  idea  expressing  how 
something  might  be  done  that  may 
eventually  lead  to  an  accepted 
procedure.  Concept  development  begins 
with  meta-analysis  and  matures  through 
an  iterative  process  employing  OR  tools. 
It  makes  assumptions  in  terms  of 
environment,  threat  and  timelines,  and  it 
determines,  describes  and  defines  the 
concept  components  in  terms  of 
DOTMLP  implications,  the  continuum 
of  military  operations,  the  relationship  to 
other  concepts  and  the  critical 
implications  for  change.  “Concept 
exploration”  expands  and  revises  the 
concept  based  on  iterative  reviews  and 
analysis.  It  can  include  seminars  and 
workgroups  and  it  typically  decomposes 
components  to  identify/validate 
questions,  issues  and  high-level 
performance  measures.  “Concept 
refinement”  expands  and  iteratively 
revises  the  concept  through  limited 
objective  experiments  to  resolve  issues, 
and  can  include  integrated  simulation 
experiments  and  integrated  field 
experiment(s).  “Experimentation 
strategy”  development  expands  and 
questions  the  issues  that  underpin  the 
concept  and  prioritizes  the  focus  for 
design  of  experiments  and  related 
events,  and  sets  the  scope  and  bounds  of 
the  design. 


Preparation  and  Conduct 

The  key  to  successful  execution  is  early 
identification  and  scoping  of  the 
objective  of  the  experiment,  related 
issues  and  hypotheses.  J94  Operations 
Division  is  staffing  a  comprehensive 
cradle-to-grave  task  list  for  the 
preparation  and  conduct  of  joint 
experiments  to  address  the  following; 

1 .  An  objective  review  process  that 
includes  resource  implications  is 
essential  for  prioritizing  issues  and 
hypotheses. 

2.  Resources  and  timelines  must  be 
identified  early  on  and  be  realistic. 

3.  Scenario  development  must 
accommodate  the  stated  hypotheses 
and  receive  CINC  approval  and 
Services’  buy  in. 

4.  Senior  leadership  should  be  briefed 
early  in  the  planning  process  on  the 
objectives  and  scope  of  the 
experiment  to  right-size 
expectations. 

5.  In  a  model-exercise-model 
experiment,  sufficient  time  must  be 
provided  for  the  pre-  and  post¬ 
exercise  modeling  activities. 

6.  Plans  must  be  established  for 
deployment  and  setup  of  the 
experiment  infrastructure  and  its 
maintenance  during  execution, 
paying  special  attention  to  the  care 
and  feeding  of  experimenters, 
computers  and  communications. 

7.  Data  collection  requirements  must  be 
developed  6-9  months  prior  to  the 
experiment  in  order  to  permit 
development  of  long  lead-time 
instrumentation  and  automated 
collection  tools. 

8.  Database  development  must  be 
accomplished  early  by  analysts  and 
modelers  working  together. 
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9.  An  end-to-end  pilot  test  of  the 
supporting  infrastructure  to  collect 
and  process  the  data  must  be 
conducted  before  start  of  the 
experiment. 

10.  Sufficient  time  to  conduct  surveys 
and  interviews  of  unit  leadership 
must  be  coordinated  early  to  ensure 
leader  availability  during  and 
immediately  after  execution. 

1 1 .  Visitor  protocol  should  be 
established  through  a  resourced 
visitor  bureau  to  ensure  proper 
treatment  of  VIPs  and  odier  visitors 
and  to  properly  manage  expectations 
of  all  concerned. 

12.  A  hot- wash  process  during 
experiment  execution  must  be 
established  to  provide  adequate  daily 
feedback  to  senior  leadership  and 
capture  insights  while  fresh. 

13.  Analysis  and  reporting  of  the 
experiment  should  be  conducted 
through  post-event  analysis 
workshops  to  include  a  wider 
analytic  conummity  in  the  final 
analysis. 

Design  of  Experiments 

Proper  scoping  and  attention  to 
managing  the  uncontrollable  and 
confotmding  variables  are  the  keys  to 
success.  The  experimental  design  must 
focus  upon  the  collection  of  the  data 
necessary  to  support  the  objectives  of  the 
experiment  whether  they  are  discovery, 
hypothesis  testing  and/or  demonstration. 
A  Full  spectrum  Analysis  employing 
Modeling  techniques,  tools  and 
Experiments  (FAME)  needs  to  be 
utilized  in  the  pre-experiment  phase  to 
focus  the  effort  by  identifying  critical 
variables,  the  ranges  of  interest  and  the 
potential  confounding  effects  as  an  input 
to  the  design  process.  Control  is 


inherently  difficult  to  achieve  due  to  the 
large  number  of  systems,  processes  and 
events  that  are  an  integral  part  of  Joint 
Experiments.  Statistical  control 
techniques  should  be  employed  to 
mitigate  this  situation.  The  effects  of 
learning  need  to  be  recognized  and 
accounted  for  in  the  design  of  the 
experiment.  The  design  of  the 
experiment  should  explicitly  consider 
the  analyses  that  are  to  take  place  in  the 
post-experiment  phase  so  that  small 
sample  sizes  can  be  “augmented”  by 
analysis,  model  results  and  simulation 
runs. 

Modeling  and  Simulation 

A  suite  of  models,  including  federations 
of  models,  needs  to  be  brought  to  bear. 
Different  models  will  be  appropriate  in 
each  of  the  phases  of  an  experiment.  The 
current  state  of  the  art  of  modeling  and 
simulation  does  not  support  the 
representation  of  ill-defined  or 
imderstood  processes  (e.g.,  cognitive 
decision  processes,  chaotic  behavior, 
information  operations).  Decision 
makers  need  to  be  educated  on  the 
limitations  of  modeling  and  simulation 
in  general  and  on  the  assumptions  and 
limitations  of  specific  models  so  that 
they  can  better  imderstand  the  results.  In 
these  cases  modeling  can  be  used  to 
capture  knowledge  as  it  is  developed  and 
to  express  hypotheses.  It  is  important 
that  the  right  model  be  selected  for  a 
particular  task.  Explicit  selection  criteria 
should  be  developed  and  employed  for 
this  purpose.  The  need  for  model 
credibility  is  a  driver  in  the  selection  of 
which  model(s)  to  use.  Currently  it  is 
difficult  to  gain  Joint  acceptance  for  a 
model  developed  in  one  Service.  It  is 
important  that  we  develop  a  joint 
approach  for  validating  models  that 
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represent  the  future  for  use  in  Joint 
Experimentation. 

Assessment  and  Utilization  of  Results 

The  value  of  an  experiment  depends,  in 
large  measure,  upon  the  technical  quality 
of  the  assessment,  the  credibility  of  the 
assessment  process  and  the 
“actionability”  of  its  results.  A  quality 
control  process  with  “checks  and 
balances”  is  needed  to  ensure  quality. 
Experiments  are  not  tests  and  a 
distinction  needs  to  be  made  between 
assessment  with  a  small  “a”  and 
assessment  with  a  large  “A.”  The 
former  being  an  “enterprise”  approach  to 
assessment  that  is  appropriate  for 
experimentation  vice  the  latter 
“independent  test”  mindset.  In  the  small 
“a”  approach  the  assessment  team  is 
fully  integrated  into  the  experimental 
planning,  design  and  conduct.  Given  the 
importance  of  assessment  to  the  overall 
Joint  Experimentation  process,  adequate 
time  and  resources  are  needed  to  get  the 
most  out  of  each  experiment  and  results 
need  to  be  documented  in  a  way  that 
facilitates  their  utilization.  An  important 
aspect  of  assessment  is  the  ability  to 
produce  very  rapid  feedback  concerning 
progress  on  experimental  objectives.  To 
ensure  that  the  need  for  speed  does  not 
degrade  the  quality  of  any  “quick-look” 
assessment,  the  assessment  team  must 
develop  a  plan  and  a  capability  for 
accomplishing  timely  preliminary 
assessments  and  to  educate  decision 
makers  and  participants  regarding  the 
caveats  associated  with  these 
preliminary  assessments.  A  “spiral” 
assessment  process  is  recommended  that 
tailors  assessment  activities  to  the 
various  stages  of  experimentation  (e.g., 
discovery,  h3q>othesis  testing, 
demonstrations).  This  will  ensure  that 


appropriate  methods  and  tools  used  as  a 
concept  are  taken  from  the  initial 
formulation  and  are  explored,  refined 
and  matured  to  the  point  where  they  can 
become  a  fielded  operational  capability. 

The  Way  Ahead 

We  have  concluded  that  experimentation 
(and  especially  large-scale 
experimentation)  is  an  essential  element 
of  any  strategy  to  transform  the  DoD 
into  a  relevant  and  capable  21^‘  century 
force.  Experimentation  is  a  valid 
construct.  If  it  is  done  well,  the  results 
will  be  more  effective  mission  capability 
packages  that  leverage  information 
superiority  to  achieve  JV2010.  Doing  it 
well  will  require  that  adequate  attention 
is  paid  to  assessment  and  validation  of 
results.  Experimentation  on  this  scale  is 
new  to  the  DoD.  It  is  not  without  its 
risks,  but  the  workshop  concluded  that 
these  risks  are  far  outweighted  by  the 
potential  of  experimentation  to  bring  out 
the  best  in  new  and  emerging  concepts, 
and  to  expedite  the  fielding  of 
significantly  improved  capabilities.  The 
pace  of  technological  change  mandates 
that  DoD  embrace  new  methods  to  move 
forward,  and  early  activity  suggests  that 
we  are  on  the  right  track  and  should  hold 
the  course. 
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Design  of  Experiments  (DoE) 
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The  WG  consisted  of  DoE  statistical  experts,  representatives  from  academia  (AFIT  and 
NPS),  senior  military  operations  research  analysts,  as  well  as  analysts  with  experience  in 
large  service  field  experiments. 
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Design  of  Experiments  (DoE) 

Key  Issues  and  Questions 

Concept  Experimentation  Strategy 

•  DoE  ^  Campaion  Design.  Not  Event  Design 

•  What  is  a  Good  Campaign  Design  (in  the  Joint 
Context)? 

-  Control  vs.  realism  of  experiment 

-  Full-Spectrum  Analysis,  Modeling  and 
Experimentation  (FAME  ):  covers  solution  space 

•  How  do  we  conduct  FAME? 

-  Continuous  design  incorporating  M-T-M-T-M... 

-  Hierarchy  or  ladder  of  Issues-Decision-Arguments- 
Hypotheses-Experiments-Measures 


DoE  is  really  a  campaign  design,  not  just  an  event  design.  So,  what  constitutes  a  good 
campaign  design  in  the  Joint  context?  We  know  what  a  good  experimental  design 
is... however,  in  the  joint  context  (i.e.,  field  exercise  events),  obtaining  a  good  experimental 
design  (e.g.,  randomized,  orthogonal,  factorial  design)  may  not  be  feasible.  So,  what  does  a 
good  design  mean  in  this  context?  And,  how  do  we  obtain  a  good  design? 

We  place  an  emphasis  on  Full-spectmm  Analysis,  Modeling  and  Experimentation  (FAME). 
FAME  is  the  complete  set  of  experiments  (wargames,  discussions,  seminars,  constractive/ 
virtual/  live  simulations,  analytic  models,  . . .),  both  before  and  after  the  field  experiment  (the 
“event”).  In  other  words,  it  IS  the  Campaign.  Design  is  not  locked  up-front,  but  evolving. 

The  campaign  design  should  flow  from  the  issues  (a  shortfall  in  capability),  which  lead  to  the 
Decisions  that  need  to  be  made  (purchase  technology  X,  implement  CONOPS  Y).  These 
Decisions  will  be  based  on  arguments  (what  is  needed  to  make  the  decision,  what  is  in 
dispute).  The  arguments  should  be  used  to  generate  hypotheses  about  something  unknown 
(technology  X  provides  improved  capability);  these  hypotheses  lead  to  experiments  which 
confirm,  reject  or  enable  us  to  refine  the  hypotheses  based  on  some  measure  (MOP,  MOE). 

One  thing  we  need  to  avoid  is  unwarranted  enthusiasm  for  the  event  results. 
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Design  of  Experiments  (DoE) 

Lessons  Learned  From  Large  Field 
Experiments  (1  of  2) _ 

•  Field  events  do  not  replace  operational  testing  requirements 

-  Large  numbers  of  systems 

-  Large  number  of  uncontrolled  variables 

-  Small  sample  sizes 

•  Late  arriving  issues  /initiatives  to  field  events  do  not  allow 
adequate  training  and  skew  data  collection 

•  A  Review  Process  is  needed  to  preclude  immature  experimental 
systems;  balanced  with  development  timelines  having  realistic 
risk  assessments 

•  Conduct  at  least  one  complete  end-to-end  rehearsal 

•  Only  one  organization  to  collect  and  process  data 


There  are  many  relevant  lessons  learned  from  service  warfighting  experiments  that  apply  to 
Joint  Experimentation. 

Experiments  provide  early  user  input  on  mature  technologies,  but  do  not  replace  operational 
testing.  The  large  number  of  systems  involved  in  the  experiment  does  not  allow 
identification  of  the  individual  system  contribution.  They  also  provide  a  large  number  of 
uncontrolled  variables  such  as  weather,  terrain,  leader  decisions,  force  mix,  etc.  The  limited 
number  of  iterations  does  not  provide  sufficient  iterations  to  determine  suitability.  “Credit” 
must  be  allowed  for  the  data  collected  during  experiments  to  reduce  future  test  costs. 

Late  arriving  issues/initiatives  do  not  allow  adequate  training  and  skew  collected  data.  In 
order  to  collect  valid  data,  personnel  involved  must  be  trained  to  effectively  employ  the 
issues/initiatives.  Adequate  MOE/MOPs  and  instrumentation  must  be  in  place  to  effectively 
collect  and  evaluate  the  issue/initiatives.  If  Subject  Matter  Experts  (SME)  are  employed  to 
collect  qualitative  data,  they  must  be  tasked,  trained  and  effectively  positioned  to  provide  the 
insights  necessary  to  evaluate  the  issue/initiative. 

A  rigorous  “Experiment  Review”  process  is  needed  to  preclude  immature  experimental 
systems;  balanced  with  development  timelines  having  realistic  risk  assessments.  Immature 
systems  require  extensive  contractor  support  to  keep  them  operational  or  must  involve 
significant  simulation.  These  technologies  disperse  the  data  collection  and  analysis  efforts. 
As  planning  for  the  large  field  experiment  progresses,  systems  not  meeting  their 
developmental  timelines  need  to  be  dropped  and  efforts  concentrated  on  the  more  mature 
technologies. 

To  insure  a  successful  experiment,  at  least  one  complete  end-to-end  rehearsal  must  be 
conducted  including  the  actual  player  units,  full  up  data  collection  and  analysis.  While,  the 
rehearsal  might  not  have  the  same  duration  as  the  experiment,  the  rehearsal  must  walk 
through  every  aspect  of  data  collection  and  analysis.  The  unit  must  be  comfortable  with  the 
instrumentation,  data  collectors  and  SMEs.  The  unit  must  be  evaluated  for  adequate  training 
on  all  issues  and  initiatives;  data  management  plans  finalized;  and  MOE/MOPs  validated. 

The  experiment  must  achieve  unity  of  data  collection.  The  unit  must  deal  with  only  one 
organization  that  will  collect  and  process  data.  The  analysts  need  one  point  of  contact  to 
optimize  data  elements  to  insure  adequate  data  is  collected.  The  data  management 
organization  must  optimize  the  data  collection  plan  to  provide  the  analysts  with  the  most  data 
with  minimal  disruption  of  unit  operations.  The  data  collection  organization  will  deliver  the 
analysts  one  consolidated  database. 
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Design  of  Experiments  (DoE) 
Lessons  Learned  From  Large  Field 
Experiments  (2  of  2) _ 

•  Big  Event  orientation  constrains  derived  vaiue 

•  Archive  raw  data  for  future  experiments 

•  Must  have  common  language.  Define  experiment, 
demonstration. . . 

•  Need  both  quantitative  and  qualitative  data  to 
conduct  full  spectrum  analysis 

•  Analysis  of  field  events  should  be  based  on 
mission  completion,  not  time  lines.  Time  should 
be  allowed  for  unanticipated  complications. 


Optimal  experimentation  requires  small,  coordinated,  well-focused  events,  but  cost 
forces  consolidation.  Smaller  experiments  will  minimize  the  deficiencies  previously 
mentioned,  but  experiments  must  have  enough  control  to  insure  valid  data.  Leveraging 
numerous  events  without  control  wastes  time  and  money.  Too  much  consolidation 
makes  analysis  extremely  difficult.  Therefore,  experimentation  is  optimized  on  cost  to 
get  the  most  data  for  the  dollar. 

All  services  must  develop  a  method  to  archive  raw  data  for  future  experiments.  Once 
the  final  report  is  written,  most  participants,  including  civilians,  move  on  to  other  tasks 
and  the  data  is  lost.  This  data  was  collected  at  great  expense  and  needs  to  be  retained 
for  future  use.  Testers  can  leverage  the  experiment  data  to  save  money.  Program 
managers  can  analyze  the  data  to  improve  the  equipment  over  time.  In  a  series  of 
experiments,  a  trend  can  be  identified  if  the  data  is  available. 

As  more  joint  experiments  are  conducted,  the  services  must  have  common  language. 
Currently,  each  service  defines  experiment,  demonstration,  etc.,  a  different  way.  Each 
definition  means  a  different  level  of  data  collection  and  analysis  effort  as  well  as 
realism.  In  order  to  coordinate  efforts  and  communicate  findings,  a  conunon  language 
needs  to  be  developed. 

To  conduct  full  spectrum  analysis,  analysts  need  both  quantitative  and  qualitative  data. 
Quantitative  data  will  tell  the  analysts  and  decision  maker  that  something  happened,  but 
qualitative  data  is  required  to  explain  why  it  happened.  This  is  especially  critical  in 
most  C4  initiatives  as  information  is  used  differently  by  each  decision  maker. 

Analysis  of  field  events  should  be  based  on  mission  completion,  not  timelines.  Models 
crash  and  cannot  support  a  command  post  exercise.  Weather  prevents  missions  from 
being  accomplished.  The  data  must  still  be  collected  to  effectively  analyze 
issues/initiative.  Redundant  events  should  be  scheduled  during  the  experiment  to  insure 
adequate  data  at  the  end  of  the  experiment.  Status  of  the  data  collection  plan  must  be 
monitored  daily  to  insure  all  the  data  requirements  are  met  at  the  end  of  the  experiment. 
If  a  certain  data  element  is  not  going  to  be  collected,  risk  assessment  must  be  conducted 
to  determine  if  the  experiment  must  be  changed  to  insure  the  appropriate  data  is 
collected.  Time  must  be  allowed  in  the  schedule  for  unanticipated  complications. 
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Design  of  Experiments  (DoE) 
Full-Spectrum  Analysis,  Modeling  and 
Experimentation  (FAME  ) 


THE  RESEARCH  AND  ANALYSIS  CAMPAIGN 
FOR  EXPERIMENTATION 


Joint  Experimentation  should  be  seen  as  a  campaign  of  events  that  uses  different 
environments  and  tools  as  a  process  for  understanding  or  discovery  of  knowledge  about 
questions  relating  to  decisions  that  will  be  made.  Although  the  campaign  will  be 
conducted  differently  for  each  experiment,  we  might  expect  to  initiate  concept 
development  through  wargames,  seminars  or  discussions  to  define  first  abstractions  and 
identify  initial  issues. 

As  the  concept  takes  form,  a  concept  model  should  be  defined  that  describes  the  processes, 
equipment,  organization,  procedures  and  information  flow  initially  required.  As  dynamic 
issues  are  identified,  the  campaign  might  move  into  studies  that  are  based  on  constructive 
simulations.  Such  simulations  are  useful  in  examining  physics-based  phenomena  in  which 
mathematical  relationships  are  known.  They  can  run  literally  millions  of  iterations 
allowing  the  investigation  of  phenomena  under  the  variance  of  selected  variables. 
Constructive  simulations  can  also  be  used  to  explore  process  definitions  and  flows  and  to 
help  bound  the  decision  space  for  virtual  and  live  experiments. 

Some  issues,  particularly  those  involving  human  performance  are  so  complex  that  they  are 
not  amenable  to  constructive  studies;  in  these  cases,  we  need  to  employ  human-in-the-loop 
methodologies.  Virtual  studies  are  particularly  important  for  examining  system  integration 
issues  where  decisions  are  integral  to  the  effective  functioning  of  systems.  Also,  there  are 
other  issues  that  require  investigation  of  a  concept  in  a  field  situation.  These  include 
larger  integration  issues  (e.g.,  the  integration  of  large  organization)  and  issues  of  stress  and 
realism  that  cannot  be  replicated  in  a  constructive  simulation  environment. 

Different  environments  and  tools  of  varying  fidelity  are  needed  because  no  one  tool  or 
environment  has  all  the  attributes  required  to  answer  all  the  questions  required  for  a 
successful  experiment.  Resource  and  safety  constraints  are  driving  factors  in  using  a 
campaign  rather  than  a  single  environment. 
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Design  of  Experiments  (DoE) 

Concept  Experimentation  Strategy 

Spiral  Flow 

Experimentation  design  is  a  continuous  process  that  should  begin  with  concept  selection  by 
the  warfighting  sponsor.  The  concepts  are  winnowed  down  to  a  select  few  that  can  be 
experimented  with.  Once  we  identify  the  concept  to  study,  we  can  determine  key 
warfighter  questions  that  must  be  answered.  Then  the  experiment  can  be  designed  (i.e., 
hypotheses,  variables,  resources,  etc.)  around  these  questions,  taking  assessment,  integration 
and  training  into  account  including  the  objectives  of  numerous  supporting  experiments.  An 
execution  plan  for  each  supporting  experiment  must  be  developed  which  may  be  a  seminar, 
wargame,  and  constructed  or  virtual  model. 

“Red  Teaming”  is  a  critical  aspect  of  the  execution  plan. 

To  react  to  experimental  discovery,  the  plan  or  the  concept  itself  must  be  revisited  after 
completion  of  each  supporting  experiment  to  ensure  proper  coverage  of  warfighters’ 
questions  and  feasibility  of  continued  experimentation.  The  designs  of  small  and  large  field 
experiments  must  draw  upon  results  of  the  earlier  experiments.  The  field  experiments, 
combined  with  earlier  experimental  results,  lead  to  action,  more  discovery  and  further 
experimentation. 

The  spiral  continues  until  the  warfighters’  questions  have  been  sufficiently  answered  —  i.e. 
the  main  concept  has  been  accepted  or  rejected.  DOTMLP  actions  must  be  assigned  to 
implement  recommended  changes  based  upon  the  experimental  results. 
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Design  of  Experiments  (DoE) 

Recommendations  (1  of  2) 


•  Focus  on  Campaign  Plan  for  Research  and 
Analysis,  not  just  Field  Experiments 

•  Campaign  Plan  Informed  by  Sense  of  Potential 

Decisions  “Model”  must  include:  Human  Wargames, ... 

•  Emphasize  Model-Test-Model  (M-T-M-T-M...) 

-  Assure  coverage  of  “scenario  space”  and 
“technical/operational  issue  space” 

-  Big  field  tests  used  for  integration,  gathering  special 
hard-to-get  knowledge,  and  demonstration 

•  Use  FAME 

•  Use  Red  Teaming  throughout 


Develop  a  campaign  plan  informed  by  a  sense  of  potential  decisions  (e.g.,  regarding  feasibility 
and  desired  nature  of  a  very-rapid  JTF  capability). 

•  Research  relevant  studies,  operational  training  data,  etc.,  to  focus  or  structure 
hypotheses. 

•  Emphasize  Model-Test-Model  (M-T-M)  paradigm  that  assures  coverage  of  “scenario 
space”  and  “technical/operational  issue  space,”  with  big  field  tests  used  for 
integration,  gathering  special  hard-to-get  knowledge  and  demonstration. 

•  For  M-T-M  to  be  real,  treat  BOGSATs,  map  games,  etc.,  as  “models”;  achieve  this  by 
structuring  the  insights  gained  from  these  techniques.  M-T-M  with  only  standard 
constructive  models  would  omit  key  sources  of  innovation  (e.g.,  the  human-in-the- 
loop). 

•  Full-spectrum  Analysis,  Modeling  and  Experimentation  (FAME).  The  purpose  of 
analysis  and  modeling  is  to  gain,  record  and  transmit  knowledge  about  the  full  space, 
not  just  to  prepare  for  and  exploit  field  experiments. 

•  Red  Teaming  is  an  integral  part  of  the  entire  process. 

•  Plan  to  assure  having,  for  “constructive”  models: 

“  Multi  resolution  model  families  —  from  closed-form  models,  spreadsheets, 
through  entity-level  simulation 

-  Interactive  modes  to  assure  innovation  and  red  teaming 

-  Decision  and  behavior  models  to  represent  alternative  human  actions  seen  in 
interactive  work  and  field  experiments. 

•  For  analysis,  plan  to  assure  having: 

-  Exploratory  analysis  in  breadth  (across  scenario  space) 

-  In-depth  analysis  where  most  needed 

-  Analysis  to  inform  improvement  of  higher-level  constructive  models  based  on 
hires  work  and  field  experiments,  and  vice  versa. 
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Design  of  Experiments  (DoE) 

Recommendations  (2  of  2) 


•  Assessment  and  Design  Groups  Must  Work 
Together 

-  Campaign  design  requires  more  then  forward 
linkage  to  assessment 

•  Seek  Top-to-Bottom  Motivation  and  Coherence 

-  Hierarchical  operational  objectives 

-  Hierarchical  functional  objectives 

•  Training:  Integral  Part  of  Campaign  Design 

-  Subjects  need  training  for  unbiased  resuits 

-  Types:  task,  technology,  procedure,  team 


•  Campaign  design  requires  more  than  linkage  to  experiment  assessment.  We  need  to 
actively  close  the  loop  on  the  M-T-M  paradigm,  i.e.,  we  need  to  fold  the  results  back  into 
the  model. 

•  Seek  top-to-bottom  motivation  and  coherence. 

-  Hierarchical  operational  objectives: 

Starting  with  DoD-specified  operational  challenges  (e.g.,  early  halt  or 
effective  intervention  in  a  1991 -Bosnia-like  situation); 

Continuing  with  subordinate  operational  challenges,  recursively  (e.g., 
establishing  immediate  theater  C2,  C4ISR,  missile  defense. . .) 

-  Hierarchical  functional  objectives: 

Starting  with,  e.g.,  global  command  and  control  architecture 

Continuing  recursively  with  subordinates  (e.g.,  worldwide  comm  suitcible 
for  all  classes  of  military  operations) 

•  Training  is  necessary  for  the  experiment  participants  to  include  task,  technology, 
procedure  and  team  training.  This  is  required  to  provide  accurate  results  from  the 
experiment  and  ensure  relevant  feedback  from  the  participants. 


15 


Design  of  Experiments  (DoE) 

Backups 
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Design  of  Experiments  (DoE) 

Agenda 

mms 

Tuesday,  9  March  1999, 1330-1700 

-  Introductory  comments 

-  Presentation/  discussion 

•  Paul  Davis,  RAND 

•  Gary  Coe,  IDA/JAWP 

Wednesday,  10  March  1999, 0800-1200 

-  Presentation/  discussion 

•  Tom  Lucas,  NPS 

•  LTC  Pete  Davidson,  OPTEC 

Wednesday,  10  March  1999, 1300-1700 

-  Presentation/  discussion 

•  Daniel  Serfaty,  Aptima 

•  Col.  Jack  Jackson,  IDA/JAWP 

Thursday,  11  March  1999,  0800-1200 

-  Presentation/  discussion  (includes  preparing  outbrief  slides) 


Slides  for  most  of  the  briefings  are  provided  in  the  appendix. 
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Design  of  Experiments  (DoE) 

Suitability  Matrix 
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ATTRffiUTES  OF  EXPERIMENTATION  ENVIRONMENTS 


Joint  Experiments  cannot  be  conducted  in  one  environment  using  one  tool;  rather  a  broad 
array  of  environments  and  tools  are  required  for  successful  experimentation. 

Hypothesis  adjudication  is  a  challenging  task  because  no  combat  model  can  be  presumed  to 
accurately  predict  the  outcomes  of  real  combat.  Indeed,  due  to  safety  restrictions  and  lack 
of  knowledge  of  future  threats,  even  live  simulations  must  be  regarded  as,  at  best,  weakly 
predictive.  Here  is  a  framework  for  understanding  how  to  use  weakly  predictive  models  for 
quantitative/  qualitative  insights  to  help  design  analysis  to  adjudicate  hypotheses.  In 
general,  hypothesis  adjudication  will  require  that  different  experiments  be  conducted  in 
different  environments. 

This  matrix  displays  the  strengths  and  weaknesses  of  several  attributes  of  experimentation 
environments.  Green  cells  indicate  that  the  environment  (row)  has  good  capability  related 
to  the  attribute  (column).  Yellow  and  red  cells  represent  moderate  and  poor  capabilities. 
This  matrix  is  an  example  of  how  to  view  a  sample  of  attributes  of  various  environments 
and  it  is  not  intended  to  represent  the  full  spectrum  of  possibilities. 

Most  events  do  not  formally  include  non-interactive  constructive  models.  Exclusion  of 
non-interactive  constructive  models  severely  restricts  the  ability  to  perform  many 
experiments,  examine  the  effects  of  many  different  variables  and  scenarios,  control  of 
nuisance  factor,  manipulate  stimuli  and  replicate  results.  Moreover,  as  constructive 
simulations  are  less  dependent  on  the  training  proficiency  of  the  solders  as  well  as 
unreliable  (as  least,  not  fully  tested)  hardware  and  software,  they  are  well  suited  to 
compensate  for  some  of  the  problems  introduced  by  training. 

The  explicit  use  of  relatively  inexpensive  environments  (wargames,  seminars  and 
constructive  simulations)  (as  compared  to  virtual  and  live  simulations)  should  be  considered 
to  supplement  the  other  test  environments  to  strengthen  the  adjudication  of  concept 
hypotheses. 
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Design  of  Experitnents  (DoE) 

The  Experimentation  Design  Cube 


Type  of  Experimentation 
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The  REASON  (front  face  of  cube)  for  developing  any  new  concept  and  entering  into  an 
experiment  can  be  categorized  as  either  attempting  to  (1)  satisfy  a  New  Mission,  (2) 
alleviate  a  known  Future  Shortfall  in  capability  —  or  discover  if  one  exists,  or  (3) 
examine  the  introduction  of  a  Technological  Opportunity. 

The  types  of  experimentation  (top  face  of  cube)  that  we  conduct  fall  into  one  of  the  three 
categories  shown.  They  are  to:  (a)  Explore  or  discover;  (b)  Demonstrate  a  capability;  or, 
(c)  test  a  proposed  Hypothesis. 

Finally,  there  are  decisions  to  be  made  based  on  the  outcome  of  the  types  of  experiments 
being  conducted  for  the  purposes  of  examining  the  concepts.  These  decisions  are  shown 
on  the  side  face  of  the  cube  as  either  (I)  Doctrinal  decisions,  (II)  Force  structure 
decisions  or  (HI)  Investment  decisions. 

This  taxonomy  divides  the  large  cube  into  27  sub-cubes  (some  of  which  may  not  be 
populated).  An  example  might  be  New  Mission-Demonstrate-Force  Structure.  This 
sub-cube  would  dictate  that  a  force  structure  decision  is  to  be  made  as  a  result  of  a 
demonstration  of  a  capability  that  will  fulfill  a  new  nnission.  We  must  now  further 
decompose  this  sub-cube  as  seen  in  the  next  slide. 
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Design  of  Experiments  (DoE) 

The  Cube  guides  Campaign  Design 


Type  of  Experimentation 


We  must  extract  this  sub-cube  and  look  at  its  dimensions.  Based  on  the  Reason,  Type  of 
Experimentation  and  the  Decision  to  be  made,  we  must  structure  our  Experiment  and  the 
subsequent  analysis  around  the  Alternatives  (front  face  of  upper  right  sub-cube).  Criteria 
(top  face  of  upper  right  sub-cube)  and  Methodologies  (side  face  of  sub-cube).  This  means 
that  the  range  and  types  of  alternatives  we  examine  in  the  experiment  are  highly  dependent 
on  the  three  factors  of  the  large  cube.  Once  the  alternatives  have  been  established,  the 
analysts  determine  the  appropriate  criteria  necessary  to  distinguish  among  the  alternatives. 
Given  these  criteria,  the  analyst  finally  determines  the  analysis  methodologies  required  to 
calculate  or  quantify  the  criteria. 

Now,  we  must  think  of  a  further  decomposition  into  a  sub-sub-cube.  This  is  represented  in 
the  cube  at  the  bottom  of  the  chart.  Once  the  analyst  determines  specific  alternatives, 
specific  criteria  and  specific  methodologies,  that  sub-sub-cube  generates  a  taxonomy  of 
Execution  Issues.  These  are  shown  to  be  Training,  Data  and  Observation.  The 
experimentation  planner  and  analyst  must  adequately  plan  for  training  the  participants  in  the 
use  of  each  alternative  (e.g.,  a  new  system  to  be  used  in  the  field).  They  must  also  plan  for 
and  execute  the  data  collection,  which  may  be  peculiar  to  the  methodology  and  criteria. 

And  finally,  the  experimenter  and  analyst  must  plan  for  and  properly  execute  the 
observation  function,  which  will  depend  on  the  data  being  gathered  and  the  training  that 
was  provided  for. 

This  taxonomy  of  cubes  and  sub-cubes  is  meant  to  highlight  the  complexity  of  the  Joint 
Experimentation  Process.  Many  times  in  the  discussions  our  Working  Group  had,  we 
would  be  talking  about  one  dimension,  then  someone  would  suddenly  take  the  discussion 
into  another  dimension  (another  cube  or  cube  face),  then  we  would  either  jump  back  to  the 
original  dimension  or  off  to  another.  We  must  understand  the  dimension  we  are  dealing 
with  in  our  discussions. 


20 


MILITARY  OPERATIONS  RESEARCH  SOCIETY 


Joint  Experimentation 
Workshop 

Planning  Experiments  Working  Group 


21 


Planning  Experiments  Work  Group 

Agenda 

•  Participants 

•  Joint/Service  Experimentation  Integration 

•  Exploration  Cycle 

•  Concept  Development 

•  OR  Contributions 

•  Process  for  Realizing  a  Concept 

•  Resource  and  Infrastructure  Considerations 

•  Interesting  Things  Discussed 

•  Issues  and  Observations 


Planning  is  the  most  critical  piece  of  any  experiment  from  an  analytic  standpoint. 
Planning  sets  the  framework  for  what  can  and  cannot  be  accomplished  analytically  to 
assist  decision  makers  with  critical  decisions.  The  steps  identified  in  this  agenda  will 
help  establish  a  well  structured  planning  process. 
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Planning  Experiments  Work  Group 

Work  Group  Participants  (1  of  2) 

• 

COL  Steven  Myer 

TRAC  Chair 

• 

COL  Rod  Reay 

AFOTEC/OL-VA  Co-Chair 

• 

Mr  Ed  Ashley 

SAIC 

• 

Mr  Grant  Ayres 

USACOM  J92 

• 

Mr  Joe  Bonnet 

SETA/Joint  Staff,  J-8-LLWAD 

• 

Mr  Mike  Borowski 

MITRBJT&E  Program  Office 

• 

Mr  Carl  Cafiero 

SYNERGY  Inc. 

• 

Mr  Bernard  Ferguson 

JT&E  PMO  Support/SAIC 

• 

Mr  Bob  French 

AF  Experimentation  Office 

• 

Mr  Jerry  Horton 

EDO 

• 

Mr  Dave  Ingle 

JBC/SAIC 

Participants  in  the  Planning  Experiments  Work  Group  brought  a  wide  range  of 
experience  in  experimentation  and  analysis  to  our  discussions. 
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Planning  Experiments  Work  Group 

Work  Group  Participants  (2  of  2) 

• 

Mr  Joe  Jackson 

MITRE/USACOM 

• 

Mr  Joe  Jennings 

MITRE/USACOM  J9 

• 

Mr  Frank  Mahncke 

JWAC 

• 

Mr  Bill  McDonald 

SPARTA  Inc. 

• 

Dr  Jimmie  McEver 

RAND 

• 

Mr  Jack  Moffatt 

AF  Experimentation  Office 

• 

MAJ  Tim  Moore 

AF  Experimentation  Office 

• 

MAJ  Robbie  Mosley 

OASD  (C3I) 

• 

Dr  Russell  Richards 

MITRE/USACOM  J9 

• 

Mr  Chuck  Walters 

JT&E  Program  Office  (OSD) 

• 

Dr  Robert  Worley 

IDA 
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Planning  Experiments  Work  Group 

Theology 

Off  Limits  Area 


Our  work  group  tried  to  stay  out  of  the  theological  area  of  discussions,  although  at  times 
we  were  unsuccessful.  We  focused  on  the  linkage  required  between  service  experiments 
and  joint  experiments.  This  planning  process  assumes  that  Joint  Experimentation  will 
leverage  existing  service  activities  for  the  near  term. 
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Planning  Experiments  Work  Group 


Joint  and  Service  Experimentation  Environment 


Joint 


Service 


JFC 

Operational 

Consensus 


Leverage 
Integration  & 
Interoperability 


Service  Component 
Service  Competencies 
Domain  Control 


■V 


Joint  Force  Capability 


•  Transition  ‘99  -  ‘01  Chasing  the  Train 

•  Steady  State ‘02  - ‘05  Driving  the  Train 


Our  group  dealt  at  length  with  determining  the  overall  objective  of  joint 
experimentation  in  relation  to  service  experimentation.  This  chart  depicts  Joint 
Experimentation  as  focused  on  the  Joint  Force  Commander  and  operational  level  issues 
conducted  through  a  consensus  process.  Service  experimentation  as  focused  on  Service 
competencies  in  warfighting  with  Title  X  domain  control.  Between  the  two  are 
leveraged  experimental  opportunities  that  have  particular  attractiveness  in  terms  of  joint 
interoperability.  Together  these  combined  experimental  efforts  should  enhance  the 
Joint  Force  Capability.  The  point  is  to  properly  position  and  combine  the  two  for  the 
greatest  benefit. 

Additionally,  as  we  plan  experimentation,  we  have  to  leverage  existing  events  in  the 
short  term  (‘99-01)  and  present  targets  and  issues  of  opportunities  for  the  Services  to 
support  in  the  steady  state  (‘02-’05). 
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Joint  Experimentation  Definition 


Joint  Experimentation  is  an  iterative  process  of 
collecting,  developing  and  exploring  concepts  to  identify 
and  recommend  the  better  value-added  solutions  for 
changes  to  DOTMLP  required  to  achieve  significant 
advances  in  future  joint  operational  capabilities. 


Campaign  Plan 


Concept  Strategic  Plan 


Experimental  Plan 


Concepts 

Components 

Questions  and  Issues 


Experimentation  is  a  process,  not  an  event.  It  is  an  iterative  process  that  continually 
develops  and  explores  concepts  through  a  campaign  plan.  A  Campaign  Plan  consists  of 
numerous  Concept  Strategic  Plans  that  occur  during  a  designated  timeframe.  Campaign 
planning  is  synchronization  among  concepts  to  get  the  most  out  of  questions  that  need  to 
be  answered  and  to  leverage  resources.  The  Concept  Strategic  Plan,  which  is  part  of  the 
Campaign  Plan,  decomposes  concepts  into  components  which  can  be  explored  to  further 
mature  a  concept.  The  Experimental  Plan  is  developed  to  identify  the  relevant/  key 
questions  and  issues  to  be  answered.  There  may  be  several  Experimental  Plans  for  a 
single  concept. 

The  Planning  Experiments  Working  Group  focused  on  the  Concept  Strategic  Plan. 
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Exploration  Cycle 


There  has  been  a  tendency  in  experimentation  to  move  directly  from  concept 
development  to  large  scale/live  experimentation.  (Quad  I  to  Quad  IV).  This  slide 
depicts  a  way  of  envisioning  the  maturing  of  a  concept  from  definition  to  validation 
through  “issue  and  scenario  space.”  In  Quad  I  the  concept  is  defined  and  developed  in  a 
rich  scenario  space  with  all  the  potential  issues  that  compose  the  concept.  In  Quad  11  the 
concept  is  explored  in  many  scenarios  with  a  refined  set  of  issues.  In  Quad  III  the 
critical  issues  that  comprise  the  concept  are  more  rigorously  investigated  in  a  limited  set 
of  scenarios  to  establish  causal  relationships,  implications,  etc.  In  Quad  FV  the  issues 
that  make  up  the  concept  are  integrated  and  validated  in  a  rich  environment  that  reveals 
the  2nd  and  3rd  order  effects  of  the  various  dynamics  in  the  concept.  Finally,  the  mature 
concept  is  modeled/investigated  constructively  in  additional  scenarios  (back  to  Quad  I). 

This  is  an  iterative  process  where  concepts  return  to  the  initial  stages  of  development  if 
not  mature  enough  to  continue.  Keys  to  this  process  are  well  thought  out  decision 
criteria/  decision  points  to  proceed  to  the  next  level. 
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Concepts 


•  Definition:  Notion  or  statement  of  an  idea  expressing  how 
something  might  be  done  or  accomplished  that  may  lead  to  an 
accepted  procedure 

•  Development 

-  Meta-analysis 

-  Iterative  process 

-  Assumptions 

-  Concept  components  and  interaction  across  DOTMLP  and 
continuum  of  military  operations 

■  Refinement/Exploration 

-  Expand  and  revise  ITERATIVELY 

-  DECOMPOSITION  of  concept  components 

•  Experimentation  Strategy 

-  Questions/  Issues  that  underpin  the  concept 

-  Focus  for  design  of  experiments/  events 

-  Scope/ Bound  for  design 


A  concept  begins  with  a  notion  or  statement.  It  is  developed  to  determine,  describe  and 
define  the  concept  components  in  terms  of  DOTMLP  implications,  the  continuum  of 
military  operations,  relationships  to  other  concepts  and  critical  implications  for  change. 

It  begins  with  a  Meta-analysis  and  matures  through  an  iterative  process.  Assumptions  in 
terms  of  the  environment,  threat  and  timelines  are  developed. 

It  is  refined  through  exploration  (seminars,  workgroups,  etc.)  and  is  expanded  and 
revised  based  on  iterative  reviews  and  analysis.  Concepts  are  decomposed  into 
components  to  identify/  verify  questions,  issues  and  upper  level  performance  measures. 

An  experimentation  strategy  is  developed  to  provide  the  focus  in  order  to  design  an 
experiment  or  event.  It  identifies  the  key  questions  required  to  underpin  the  concept  and 
provide  the  answers  which  will  inform  the  validity  and/or  value  added  of  a  concept.  The 
experimentation  strategy  provides  the  scope  and  bounds  of  an  experiment  in  terms  of 
DOTMLP,  the  continuum  of  military  operations,  component  capabilities,  areas  of 
greatest  potential  and  the  baseline. 
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OR  Management  Tools 


•  Baseline  Assessment  Database 

•  Concept  Hierarchy  Tool 

•  Events  Database 

•  Issue  Integration 

•  Task  Analysis 

•  Concept  Prioritization/  Cost  Benefit 


In  addition  to  the  Operations  Research  (OR)  tools  and  techniques  that  are  available  for 
experimentation,  there  are  other  tools  to  manage  the  experimentation  process  and 
experimentation  campaign  itself  J9  has  already  developed  a  suite  of  applicable  tools  to 
do  this  as  listed  in  this  slide. 

An  area  for  further  development  is  task  analysis.  As  the  UJTL  becomes  a  more  critical 
and  matured  framework  to  investigate  force  capabilities,  task  analysis  and  management 
through  the  task  analysis  framework  will  be  a  critical  OR  capability. 
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Exploration  Techniques 


Spectrum  of  Evaluation  Techniques  and  Environments 


Technique 

Resources 

Lead  Time  to 
Create  and 
Apply 

Breadth  of 
Application 

Replicability 
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Very  Broad 
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Low 

Weeks/Davs 

Very  Broad 

Limited 

Fair 

Analytical 

Models 

Low 

Weeks/Days 

Broad 

Fully 

Fair 

Lab 

Experiment 

Low- 

Moderate 

Weeks/Days 

Limited 

Fully 

Moderate  to 
Good 

Constructive 

Simulation 

Low- 

Moderate 

Months/Days 

Broad 

Fully 

Moderate 

Virtual 

Simulation 

High 

Years/ 

Months 

Moderate 

With 

Difficulty 

Potential  for 
Good 

Live 

Simulation 

High 

Years/Weeks 

Limited 

Little 

Generally 

Good 

Real  Crises/ 
Combat 

N/A 

N/A 

Quite  Limited 

None 

Excellent 

This  chart  shows  possible  exploration  techniques  and  environments  available  for 
experimentation  in  terms  of  resources  and  lead  time  required,  the  breadth  of  the 
application,  replication  capability  and  the  credibility  associated  with  a  technique.  Each 
technique  can  be  used  by  itself  or  in  combination  with  any  other  technique. 
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Framework 
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Constructive/Virtual  Simulation 
Wargames 


This  chart  recommends  a  framework  for  concept  development  and  experimentation.  It 
provides  an  example  of  an  iterative  process  and  how  different  types  of  analyses  and 
events  can  be  structured  to  support  a  concept.  Again,  it  is  important  to  note  that  this  is 
an  iterative  process  and  at  any  decision  point  a  concept  may  be  terminated,  go  back 
single  or  multiple  steps  or  go  straight  to  a  recommendation. 

Combined  with  the  concept  development  slide,  this  framework  provides  a  structure  to 
develop  the  Concept  Strategic  Plan  as  well  as  a  multi-year  Experimentation  Campaign 
Plan. 
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Resource  and  Infrastructure  Consideration 


•  Scenarios 

•  Tool  Development 

•  Red  Teaming/  Adversary  Development 

•  DOTMLP  Development/  EXFOR 


Resources  and  infrastructure  are  important  considerations  throughout  the  entire  planning 
process.  Scenarios  must  be  developed  to  provide  the  analytic  rigor  in  which  a  concept 
will  be  explored.  Tools  (i.e.  models  and  simulations)  must  be  carefully  considered  in 
order  to  provide  answers  to  issues  and  support  MOE  and  MOP.  Models  and  simulation 
capabilities  must  also  be  considered  as  drivers  of  live  events.  Red  teaming  and/  or 
opposing  force  capabilities  must  be  developed  to  provide  robust  and  credible  adversaries 
in  order  to  explore  concepts. 
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Interesting  Things  Discussed  Not  Resolved 


•  Concept  Selection  Process 

•  Campaign  Planning 

•  Questions  and  Issues  ◄ — ►  Events 

•  Taxonomy  Development 


The  Planning  Experiments  Working  Group  touched  on  the  following  topics  but  did  not 
focus  on,  or  resolve,  any  issues  related  to  these  topics.  In  short,  methodologies  need  to 
be  developed  for  the  selection  of  concepts,  the  planning  of  campaigns  and  the  linkage  of 
concept  questions  and  issues  with  events  or  venues  for  exploration/  experimentation. 
Joint  taxonomy  must  be  developed  for  better  understanding  of  joint  experimentation 
among  the  services. 


34 


Planning  Experiments  Work  Group 

Recommendations 


•  OR  Techniques  and  Analysis  be  applied  to  Concept  Definition  and 
Selection. 

•  Campaign  Plan  ‘00  have  an  enhanced  articulation/common 
understanding  of  the  JE  Vision  and  end  state  we  are  pursuing. 

•  Use  of  operations  research  tools  and  techniques  be  planned  and 
integrated  throughout  the  Concept  Development/  Exploration  cycle. 

•  The  current  J9  Organization  for  Experimentation  be  reviewed  to  provide 
continuity  in  view  of  Concept  Development,  Investigation  and 
Implementation. 

•  The  Campaign  Plan  develop  criteria  and  a  decision  process  to  make 
hard  choices  in  time,  resources  and  appetite. 

•  Low  Resolution  and  Interactive  Development  be  planned  before  Live 
Events  and  as  part  of  an  Model-Exercise-Model  methodology. 

Dick,  should  these  recommendations  read  as  “should  be  applied,”  “should  be  planned,”  etc.  ??  Corrina 


The  process  of  concept  definition  and  selection  requires  thoughtful  investigation  and  analysis 
and  should  have  its  own  planned  assessment  process.  Concepts  should  be  prioritized  LAW  the 
JE  vision. 

There  does  not  appear  to  be  a  common  understanding  of  what  the  results  of  JE  are  supposed  to 
be.  Without  this  vision  it  is  impossible  to  prioritize  effort  and  measure  experimental  progress. 

Assessment/analysis  does  not  happen  at  the  end  of  the  experimental  process,  it  is  integral 
to  it  Some  of  the  highest  payoff  in  assessment  will  be  early  in  the  concept  exploration  phase 
and  must  be  planned  and  resourced.  Analytic  perspective  must  be  embedded  throughout  the 
process,  especially  planning.  Current  J9  staff  is  long  on  planners  and  short  on  analysts.  In  its 
final  configuration,  J9  must  have  staff  with  an  analytic  capability  involved  in  every  aspect  of 
experimentation. 

The  current  J9  organization  follows  the  tradition  J-staff  logic,  but  may  not  be  appropriate  for 
experimentation.  With  an  iterative,  concept  based  approach  it  might  be  better  to  organize 
around  concepts  rather  than  functionally.  This  idea  should  be  reviewed  at  the  J9  off-site. 

Currently  there  is  a  “take  all  concepts”  approach  to  JE.  To  do  things  right,  they  must  be 
focused.  A  process  to  make  choices  and  prioritize  should  be  part  of  the  Campaign  Plan  and 
involve  analysis  techniques  to  support. 

Model-Exercise-Model  is  a  must.  There  is  high  Return  On  Investment  (ROI)  at  the  concept 
exploration  part  of  the  process  that  analysis  can  richly  inform. 
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MILITARY  OPERATIONS  RESEARCH  SOCIETY 


Joint  Experimentation 
Workshop 

Conduct  of  Experiments 
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Preparation  and  Conduct  of  Experiments 

Working  Group  Tasks 

•  Provide  ACOM  Joint  Experiment  Directorate  (J9) 
with: 

-  Current  Issues 

-  Innovations 

-  Future  Trends  to  Consider 

•  In  the  following  areas: 

-  Exercise  preparation 

-  Experiment  execution 

•  Where  the  rubber  (design)  meets 
the  road  (experimental  unit) 


Many  individuals  in  and  out  of  the  government  currently  associated  with  the  services 
battle  labs  and  Tc&E  agencies  have  years  of  experience  in  conducting  separate  service 
experiments  or  overlaying  an  experimental  design  on  a  real-world  exercise  or  event. 
Participants  in  this  work  group  extrapolated  from  these  experiences  to  generate  lessons 
learned  concerning  the  preparation  and  conduct  of  joint  experiments  outside  the 
laboratory. 
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Preparation  and  Conduct  of  Experiments 

Working  Group  Participants 


•  Dr  Rick  Kass  Army  Test  and  Experimentation  Command 

(TEXCOM) 

•  Mr  Don  Kroening  Army  TRADOC  Analysis  Center  -Leavenworth 

(TRAC-L) 

•  COL  Rick  Geraci  ACOM  Joint  Experimentation  Directorate-J9 

(Operations  Division-J94) 

•  Participants 

•  Mr  Gary  Arnett  -  Synergy  Inc 

•  Dr  Thomas  Cook  -  Army/ARL 

•  Dr  Richard  LaFerriere  -  Army/TRAC 

•  Ms  Leslie  Lampella  -  Army  /HQ  TRADOC 

•  LTC  Lynnette  Lowrimore  -  AF/XOCI 

•  LTC  Chris  Pate  -  DCSOPS 

•  Ms  Lauran  Winter  -  Army/HQ  TRADOC 

•  Mr  Dale  Winzer  -  AF/AC2ISRC/AFEO 


The  input  to  the  working  group  reflects  great  Army  and  Air  Force  experience.  One  of 
the  civilians  was  a  retired  Marine  Corps  officer  so  he  provided  both  Marine  Corps  and 
Navy  perspectives. 
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Preparation  and  Conduct  of  Experiments 

Preparation 


•  Identify  and  scope  experimentation  hypotheses  eariy  — 
key 

“  All  else  depends  on  this 

“  Need  objective  way  to  review  and  prioritize  hypotheses 

•  Relevance,  technical  maturity,  scenario  impact,  cost,  modeling  feasibility 
and  data  requirements 

•  Eariy  pianning  document  deveiopment  and  staffing 

-  Need  early  initial  Management  Plan 

•  Primary  issues,  scope,  and  concept;  roles,  missions  and  responsibilities  of 
partners;  product  timelines 

•  Eariy  identification  of  resource  requirements 

-  Initial  and  living  budget 

“  Outside  personnel  requirements  very  early  (data  collectors,  SMEs) 

-  Consider  contracting  out  to  existing  OTAs  who  do  this  as  a  way  of  life 


The  key  to  a  successful  experiment  is  the  early  identification  and  scoping  of  the  hypotheses 
(analytic  issues  to  be  addressed  in  the  experiment).  Since  the  experiment  is  such  a  visible 
event,  there  will  be  many  agencies  wanting  to  use  it  for  their  own  purposes.  Prioritization 
of  the  hypotheses  can  be  accomplished  in  a  series  of  review  boards  that  apply  specific 
review  criteria  to  determine  the  prioritization  of  the  individual  hypotheses.  These 
prioritization  criteria  should  include  areas  pertinent  to  the  cost  and  ability  to  examine  the 
hypothesis  in  the  M&S  and  the  field  experiment  itself. 

Early  planning  documents  are  important  because  they  provide  a  common  understanding  of 
the  experiment  to  all  involved,  A  key  planning  document  is  an  early  Experiment  Directive 
(ED)  which  assigns  responsibilities  and  defines  the  timelines  for  the  experiment.  This 
document  must  be  developed  early  in  the  process  and  signed  by  a  senior  official  with  the 
authority  to  assign  responsibilities.  This  overarching  management  directive  can  then  be 
followed  by  a  more  detailed  plan  for  the  experiment  which  includes  sub-plans  or  annexes 
for  the  management  of  the  experiment,  the  M&S  used  in  the  experiment,  the  data  collection 
plan  and  any  other  plan  appropriate  for  the  experiment. 

Resource  needs  are  shaped  by  the  initiatives  and  hypotheses  of  the  experiment.  Start  as 
early  as  possible  and  work  actions  in  parallel  to  resource  long-lead  items  —  especially 
personnel  resources  required  from  external  agencies.  For  large  Joint  Experiments  consider 
“contracting”  the  data  collection- data  management  to  an  existing  DoD  agency  with 
expertise  in  running  tests  and  experiments,  for  example,  to  one  of  the  service  Operational 
Test  Agencies  (OTA).  They  have  an  infrastructure  of  instrumentation  and  data 
management  experience. 
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Preparation  and  Conduct  of  Experiments 

Preparation 


•  Scenario  development 

-  Must  accommodate  experimental  hypotheses 

-  Need  CINC  approval  and  service  buy-in 

-  Internal  verification  and  external  validation,  iterative  refinement 

•  M&S 

-  Need  to  schedule  sufficient  time  for  modeling  cycle 

-  If  experiment  =  model-exercise-model  then  it’s  not  over  until  post¬ 
exercise  modeling  component  completed 

-  Collect  additional  data  during  exercise  to  support  calibrating  post¬ 
exercise  modeling 

•  Deployment  and  set-up 

-  Transportation  -  budget,  priority,  mode 

-  Bed  down  and  site  set-up  -  personnel,  logistics,  communication  checks 


New  joint  scenarios  will  be  required  if  the  joint  concept  is  truly  new.  Joint  scenarios  will 
need  to  be  approved  at  the  highest  level  and  coordinated  early  to  get  everyone  to  buy  in. 

The  VVA  process  will  need  to  be  done  in  parallel  because  of  shortened  timelines.  The 
scenario  refinement  process  for  M&S  and  the  field  experiment  will  continue  throughout  the 
planning  stage  and  into  the  conduct  of  the  experiment  itself. 

Develop  the  M&S  plans  early.  M&S  needs  a  long  lead  time.  Late  breaking  “good  ideas” 
adversely  impact  M&S  development.  Sufficient  time  after  the  field  exercise  needs  to  be 
allocated  to  complete  the  post-exercise  M&S.  Of  special  consideration  are  those  M&S  data 
requirements  that  need  to  be  collected  during  the  field  experiment  for  input  into  the  post¬ 
exercise  M&S  to  support  the  final  assessment  phase. 

Getting  experimenters  (experiment  controllers,  data  collectors,  analysts,  etc.)  and 
participants  (units  and  augmentees  who  play  in  the  experiment)  to  the  various  events  will  be 
a  major  consideration.  USACOM  must  budget  and  POM  for  CJCS  transportation  dollars  to 
cover  all  planning,  training  and  spin-up  events,  as  well  as  the  culminating  event.  Priority  at 
least  as  high  as  CJCS-sponsored  exercises  must  be  accorded  the  joint  experimentation 
infrastructure. 

Responsibility  for  bed  down  of  experimenters  and  participants  may  be  delegated  to  the 
service  components,  but  USACOM  must  set  the  ground  rules,  manage  the  budget,  and 
ensure  that  safety,  security,  local  transportation,  billeting,  messing,  communications,  etc.  are 
sufficient  to  support  the  event. 
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Preparation  and  Conduct  of  Experiments 

Preparation 

•  Data  collection  planning 

-  Develop  data  collection  requirements  6-9  months  early 

•  Instrumentation  long  lead  item 

•  Transparent  data  collection  techniques  (automated  tools) 

•  Quantitative  and  qualitative 

-  Establish  early  experimental  unit  leadership  availability 
requirements  for  surveys/interviews 

-  Early  data  base  development  with  analyst  and  modeler 

-  Pilot  test  end-to-end!!!! 


At  least  9  months  prior  to  the  experiment  the  analysts  must  provide  the  data  collection 
organizers  with  collection  requirements.  The  collection  requirement  will  dictate  the 
number,  type  and  qualifications  of  SMEs/data  collectors  and  the  quantity  and  type  of 
instrumentation  needed.  Transparent  data  collection  techniques  along  with  automation 
should  be  used  as  much  as  possible.  Use  manual  data  collection  only  when  absolutely 
necessary. 

Negotiations  with  the  actual  experimental  unit  are  also  necessary  so  that  unit  leaders  and 
key  players  in  the  experiment  will  be  available  for  in-depth  interviews  and  surveys. 

A  database  that  meets  the  needs  of  both  the  collection  agency  and  the  analysis  agency  also 
needs  early  development. 

Finally,  time  must  be  allocated  for  a  pilot  test  or  an  end-to-end  test  prior  to  the  start  of  the 
exercise.  This  will  ensure  that  the  collection  procedures  are  adequate  and  that  the  analysts 
are  satisfied  with  the  data  before  the  actual  exercise  starts. 
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Preparation  and  Conduct  of  Experiments 
Preparation 

•  Training 

-  Data  collectors,  SMEs,  experimental  force, 
OPFOR 

•  Smart  book,  hypotheses  card 

-  Senior  leadership 

•  Right-size  expectations, 

-  Issue  priority,  quaiity  of  data,  resource 
requirements/rationale 


A  good  method  for  getting  a  common  reference  for  everyone  involved  in  the  execution 
of  the  experiment  is  to  develop  a  “smart  book”  and  a  “hypothesis  card.”  The  smart  book 
is  specifically  designed  for  SMEs  to  carry  and  it  identifies,  describes  and  explains  all  of 
the  hypotheses/issues/initiatives  in  the  experiment.  The  hypothesis  card  is  a  laminated 
business-size  card  that  succinctly  summarizes  the  goals  and  objectives  of  the 
experiment.  One  hundred  percent  of  the  participants  (to  include  data  collectors,  SMEs, 
player  unit  and  OPFOR)  should  be  required  to  carry  this  card.  It  does  not  reflect  well  to 
have  a  VIP  visit  the  site  and  ask  one  of  the  participants  what  they  are  involved  in  —  if 
the  participant’s  response  is  a  shrug. 

The  senior  leadership  in  both  the  CINC  and  service  staffs  need  to  be  educated  on  the 
scope  and  goals  of  the  experiment.  This  can  be  accomplished  by  a  traveling  “road 
show.”  One  goal  in  this  education  process  is  right-sizing  expectations.  Another  goal  is 
to  convince  organizations  supporting  the  experiment  that  their  contributed  resources  are 
being  used  in  a  great  cause  and  that  their  continued  support  is  required  and  appreciated. 
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Preparation  and  Conduct  of  Experiments 

Conduct 

•  Data  collection  execution 

-  Provide  quick-look  capability  (manual  and  instrumented)  to  insure 
getting  required  data  and  to  support  daily  hot-wash 

-  Analysts  access  to  experiment  execution 

-  Ready  response  cell  of  SMEs/data  collectors 

•  Control  of  experiment 

-  Need  a  robust  control  cell  that  Includes  analyst  representation 

-  Balance  free  play  and  MSEL  events 

•  Visitor  protocol 

-  Resource  visitor  bureau  —  pays  dividends  (. . .  but  non¬ 
interference) 

•  Standard  briefingAours  (to  include  into  experimental  unit),  opportunity 
to  right-size  expectations,  send  out  invitations,  VIP  treatment 


It  is  not  a  good  idea  to  wait  until  the  experiment  is  completed  to  determine  if  adequate  data 
was  collected.  Analysts  need  to  ensure  that  the  data  is,  in  fact,  coming  in  as  the  experiment 
progresses.  This  allows  for  data  collection  adjustments  while  the  experiment  is  in  progress. 
Analysts  must  occasionally  get  away  from  their  computer  printouts  to  view  the  experiment 
first  hand  and  to  get  their  own  views  of  what  is  happening  in  order  to  better  understand  the 
data.  Sometimes  collection  requirements  change  during  execution.  Additional  qualified 
SMEs  need  to  be  omhand  to  fill  the  gaps  without  diminishing  other  collection  requirements 

The  exercise/experiment  control  cell  needs  to  include  analysts  so  both  “impact  on  data 
collection”  and  “impact  on  experimental  objectives”  will  be  considered  when  contemplating 
scenario  adjustments.  The  experimenter  will  always  have  to  balance  experimentation  goals 
(leaming/discovery)  and  exercise  goals  (training).  Can  both  co-exist  in  one  event  without 
diluting  the  desired  results  due  to  competing  objectives  and  requirements?  Free  play  is 
desired  but  an  experiment  must  generally  be  more  structured  than  a  training  exercise. 

Telling  the  experimentation  story  well  is  an  absolute  necessity.  Experiments  are  expensive 
propositions.  Senior  leaders  (service,  joint  and  DoD)  as  well  as  Congress  want  to  ensure  the 
money  is  well  spent.  This  political  reality  can  not  be  ignored.  Leaders  will  often  want  to  see 
the  experiment  first  hand  to  validate  that  the  money  is  well  spent.  Thorough  planning  for 
visitors  is  critical  to  the  success  of  the  experiment.  Visitor  briefings  are  also  an  opportunity 
to  further  manage  expectations. 
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Preparation  and  Conduct  of  Experiments 

Conduct 

•  Maintain  experiment  infrastructure 

-  Care  and  feeding  of  experimenter 

-  Establish  computer  maintenance  and  separate  communication  networks 

•  Hot  wash  briefing  requirements 

-  Don't  fight  appetite  for  immediate  feedback 

•  Establish  process  to  support  generating  daily  results  briefings 

•  Use  process  to  capture  insights  while  fresh 

•  Train  analysts  to  do  incomplete  analysis 

•  Select  credible  briefer  (military  analysts) 

•  Analysis  and  reporting 

-  Conduct  post-event  analysis  workshops,  don’t  develop  report  in  a  vacuum 

-  Produce  one  integrated  report  preceded  by  short,  hard  hitting  “insights” 
memo  (final  report  after  post-exercise  modeiing  —  not  after  “field 
exercise”) 

_ 


Sufficient  resources  must  be  allocated  to  keeping  the  experiment  infrastmcture  and  systems 
architectures  up  and  running  throughout  the  experiment.  Do  not  depend  on  the  player  unit 
for  computer  maintenance  and  communication  nets  to  run  the  experiment  infrastructure. 

The  experimentation  community  must  realize  that  there  will  be  a  natural  requirement  on  the 
part  of  senior  leadership  to  have  some  immediate  and  cyclic  analytic  feedback  during  the 
course  of  the  experiment/exercise.  The  experimentation  community  should  not  try  to  resist 
this  tendency,  but  rather  should  use  it  as  an  opportunity  to  update  right-sizing  expectations. 
Accordingly,  the  hotwash  must  include  positive  and  negative  findings,  and  be  grounded  in 
reality. . .it  should  not  be  a  “love-in.” 

To  facilitate  the  analysis  (given  that  much  of  it  is  being  conducted  by  groups  of  analysts 
who  are  geographically  dispersed  after  the  experimental  event),  post-event  analysis 
workshops  should  be  conducted.  The  focus  of  these  efforts  is  to  produce  two  products.  The 
first  product,  an  Initial  Insights  Memorandum  (IIM),  is  a  short  accounting  of  the  major 
insights  drawn  from  the  experiment,  even  if  produced  from  incomplete  data.  This  product 
is  intended  to  satisfy  the  immediate  appetite  for  analytic  results  by  the  senior  leadership. 

The  second  product  is  the  final  report  for  the  experiment.  This  report  must  be  a  single 
document  which  contains  all  analytic  results  of  the  experiment,  regardless  of  who 
developed  them.  A  single  report  is  necessary  to  ensure  that  contradictory  results  from 
multiple  reports  are  not  released  to  the  consumers. 
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Preparation  and  Conduct  of  Experiments 


Finding/Recommendation 


•  Finding 

-  Insufficient  attention  to  experiment  planning  and  execution  can 
result  in  an  invalid  experiment 

•  Insufficient  resources 

•  Unrealistic  Expectations 

•  Nonresponsive  data  collection 

•  Inadequate  analysis  and  reporting 

•  Recommendation 

-  Develop  a  single  integrated  J9  team  with  responsibilities  to  foilow 
the  experiment  from  concept  development  to  final  report  to  ensure  a 
valid  experiment. 


“An  experiment  without  good  data  is  just  a  training  exercise” 


It  is  difficult  to  summarize  the  collection  of  “good  ideas”  generated  from  this  working 
group.  The  difficulty  was  not  in  generating  insights  based  on  the  working  group’s 
experienced  members,  but  rather  in  organizing  the  many  good  and  varied  insights  into  a 
useable  product  for  ACOM. 

If  there  was  a  consensus  of  the  working  group  it  would  probably  have  focused  on  the 
proposition  that  the  final  stage  “analysis  of  data  to  produce  a  report”  needs  to  be 
represented  early  in  the  experiment  design  process  or  the  desired  results  will  not  occur. 
This  design  process  needs  to  be  considered  early  during  hypothesis  generation,  resource 
allocation  and  in  right-sizing  the  expectations  of  the  product  consumers. 
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Preparation  and  Conduct  of  Experiments 


Backup  Slides 
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Preparation  and  Conduct  of  Experiments 

Concept  Hypotheses 

•  Identifying  and  scoping  of  all  hypotheses  -  Key! 

•  Determine  “jointness”  of  hypotheses 

•  Need  objective  way  to  prioritize  hypotheses  - 
(Hypotheses  Review  Board) 

•  Establish  review  criteria 

-  Relevance  -  Scenario 

-  Technical  Maturity  -  Modeling  feasibility 

-  Cost  “  Data  Requirement 

•  Top-down  driven  -  early-on  senior  leader 
involvement 

•  Sensitize/explain  the  impacts  of  last  minute  add¬ 
ons 


The  key  to  a  successful  experiment  is  the  early  identification  and  scoping  of  the 
hypotheses  (analytic  issues  to  be  addressed  in  the  experiment).  Since  the  experiment  is 
such  a  visible  event,  there  will  be  many  agencies  wanting  to  use  it  for  their  own 
purposes.  A  process  must  be  set  up  to  ensure  that  the  experiment  is  limited  to 
addressing  only  necessary  hypotheses,  prioritized  by  proper  authority.  In  the  context  of 
joint  experiments,  the  extent  of  “jointness”  of  the  hypotheses  must  be  determined  and 
agreed  to.  The  prioritization  of  the  hypotheses  can  be  accomplished  by  a  formal  review 
board,  or  series  of  review  boards,  which  apply  specific  review  criteria  to  determine  the 
prioritization  of  the  individual  hypotheses.  The  prioritization  criteria  should  include  a 
hypothesis’  relevance  to  the  experiment  objectives,  the  technical  maturity  of  the  systems 
involved,  the  associated  costs,  the  ability  of  the  scenario  to  address  it,  the  feasibility  of 
M&S  to  address  it  and  the  data  required  to  represent  it  in  the  appropriate  M&S.  The 
development  of  potential  hypotheses  to  be  reviewed  and  selected  for  the  experiment 
should  be  a  top-down  process  driven  by  the  senior  leadership.  Senior  leadership  must  be 
sensitized  to  the  fact  that  changes  to  the  hypotheses  late  in  the  process  can  have 
devastating  effects  on  the  success  of  the  experiment. 
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Preparation  and  Conduct  of  Experiments 

Experiment  Planning  Documents 

•  Documents  the  core  understanding  of  the  experiment 

•  Needs  a  balance  to  adequately  address  all  of  the  consumers  of 
the  plan 

•  Experiment  directive/project  order 

-  Assigns  the  overall  concept,  roles,  mission,  responsibilities  and  product  timelines 

-  Developed  and  signed  early 

•  Best  to  have  a  single  integrated  plan  (living  document) 

-  Management  plan 
~  M&SPIan 

-  Data  collection  plan 

-  Coordination  through  intermediate  versions  is  key 
“  Define  common  terminology 

-  Coalition  considerations  should  be  addressed 

•  Right-sizing  of  expectations  -  brief  the  plan 

•  All  stakeholders  at  highest  levels  sign  plan 


Planning  documents  are  key  to  the  experiments’  success  because  they  provide  a  common 
understanding  of  the  experiment  to  all  involved.  These  plans  must  be  balanced  so  that  they 
adequately  address  all  issues  and  actors  involved  in  the  experiment.  For  example,  do  not 
have  a  50-page  operations  order  with  a  one-page  add-on  of  experiment  measures  of 
performance. 

The  key  planning  document  is  an  early  Experiment  Directive  which  assigns  responsibilities 
and  defines  the  timelines  for  the  experiment.  This  document  must  be  developed  early  in  the 
process  and  signed  by  a  senior  official  with  the  authority  to  assign  responsibilities. 

This  directive  is  closely  followed  by  a  single  plan  for  the  experiment  which  includes  sub¬ 
plans  or  annexes  for  the  management  of  the  experiment,  the  M&S  used  in  the  experiment, 
the  data  collection  plan,  and  any  other  plan  appropriate  for  the  experiment.  The  key  is  for  all 
these  planning  documents  to  be  integrated  into  a  single  plan.  This  plan  must  be  fully 
coordinated,  use  common  terminology  and  address  the  potential  inclusion  of  coalition  forces 
in  the  experiment. 

The  plan  must  be  briefed  to  senior  leadership,  to  among  other  things,  right-size  their 
expectations  regarding  the  experiment. 

This  detailed  plan  must  also  be  approved  and  signed  by  the  stakeholders  in  the  experiment 
who  have  the  authority  to  task  others  involved. 
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Preparation  and  Conduct  of  Experiments 

Resource  Management 

•  Impacted  by  initiatives/concept  issues 

•  Start  early;  never  enough  lead  time 

•  If  budget  is  set,  additional  requirements  and  consequences 
need  to  be  surfaced  as  soon  as  possible 

•  Management  plan  defines  responsibilities  and  time  lines 

•  Must  identify  person  in  charge  of  physical  site 

•  Need  a  resource  support  plan 

•  Identify  personnel  to  meet  defined  qualifications 

•  May  contract  existing  agencies  (OTAs)  who  have  experience 
with  experiments 


Resource  needs  are  shaped  by  the  initiatives  and  hypotheses  of  the  experiment  and  constrained 
by  the  budget.  There  is  never  enough  lead  time.  Start  as  early  as  possible  and  work  actions  in 
parallel.  As  additional  resource  requirements  arise  after  the  initial  budget  is  approved,  these 
UnFunded  Resources  (UFR)  need  to  be  surfaced  as  soon  as  possible  along  with  the 
consequences  of  not  resourcing  in  order  to  allow  the  senior  decision  makers  time  to  find  the 
resources.  The  management  plan  fixes  the  responsibilities  and  timelines  for  the  major 
resources. 

It  is  necessary  to  identify  the  person  in  charge  of  decisions  regarding  the  physical  site  and 
insure  that  this  person  has  authority  to  enforce  decisions.  The  infrastructure  includes  offices, 
computers,  phones,  equipment  and  support  personnel.  The  badging  system  includes  Web- 
based  registration  and  the  in-processing  system.  It  is  critical  to  keep  all  participants  informed. 

One  also  needs  a  resource  support  plan  and  a  disciplined  approach  to  resourcing,and  to 
identify  resources  for  each  phase  of  MEM.  It  is  necessary  to  develop  tasking  for  the  people 
and  equipment  to  meet  the  timelines.  Participating  units  should  provide  their  own 
infrastructure.  Floor  plans  to  identify  location  of  experimenters  and  to  assess  their  impact  on 
the  operators  are  also  required. 

Subject  Matter  Experts  (SME)  should  be  identified  by  qualifications  and  quantity  to  support 
the  experiment  as  data  collectors,  and  to  identify  equipment  they  may  require.  Task  agencies 
who  provide  SMEs  early  on.  Have  a  vetting  process  so  if  the  SME  provided  does  not  meet 
SME  qualifications  (e.g.,  rank,  combat  experience,  specialty),  he  or  she  can  be  replaced. 

For  large  Joint  Experiments  consider  '‘contracting”  the  data  collection-data  management  to  the 
expertise  of  an  existing  DoD  agency  with  expertise  in  running  tests  and  experiments,  e.g.,  to 
one  of  the  service  operational  test  agencies.  They  have  an  infrastructure  of  instmmentation 
and  data  management  experience. 
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Preparation  and  Conduct  of  Experiments 

Scenario  development 

•  Scenario  should  be  concept-driven;  needs  to 
reflect  experimental  objectives 

•  Representative  ‘Joint’  flavor 

•  CINC  approval  and  Service  buy-in 

•  Consider  both  internal  verification  and  external 
validity 

•  Iterative  refinement  through  execution 


Developing  scenarios  for  joint  operations  will  not  be  easy  given  the  newness  of  the 
endeavor  and  the  requirement  to  get  the  services  to  buy  in.  New  joint  scenarios  will  be 
required  if  the  joint  concept  is  truly  new.  The  scenarios  need  to  be  approved  at  the 
highest  level  and  coordinated  early  to  get  everyone  to  buy  in.  The  VVA  process  should 
be  done  in  parallel  because  of  shortened  timelines.  The  scenario  refinement  process  will 
continue  throughout  the  planning  stage  and  into  the  conduct  of  the  experiment  itself. 
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Preparation  and  Conduct  of  Experiments 
M&S 

•  Determine  M&S  capabilities  to  support  concept 
experimentation 

•  Develop  M&S  plan  early  (always  a  long  lead  time) 

-  Plan  M&S  support  for  all  experimentation  phases 

-  Identify  data  requirements 

-  Identify  M&S  resources 

•  Federation  of  M&S  needs  to  be  considered  for 
Joint  operations 


Determine  M&S  capabilities  to  support  concept  experimentation:  What  can  M&S  do 
and  not  do?  Can  the  issue  be  modeled?  Can  the  issue  be  simulated  during  the  field 
phase?  Can  the  data  be  obtained/collected? 

Develop  M&S  plans  early.  Late  breaking  “good  ideas”  adversely  impact  M&S 
effectiveness.  Early-on  fix  responsibility,  identify  data  requirements  and  data  inputs. 
The  data  providers,  approvers  and  arbitrators  for  this  process  need  to  be  identified  also. 
A  special  consideration  are  the  M&S  data  inputs  that  need  to  be  collected  during  the 
field  experiment  for  the  M&S  post-exercise  assessment  phase.  Identification  of 
resources  for  M&S  is  also  necessary  —  WA,  equipment,  any  unique  development, 
connectivity,  M&S  players,  security  and  floor  space. 

Also  consider  a  federation  of  M&S  to  address  all  joint  participants  and  capabilities. 
Review  appropriate  analytical  and  training  M&S  to  address  experimental  issues. 


Preparation  and  Conduct  of  Experiments 

Deploy  and  Set-up 

•  Transportation 

-  Budget  (POM,  FINPLAN,  resource  management) 

-  Priority  (establish  CJCS  priority) 

-  Mode  (air,  sea,  road-haul) 

•  Bed  down 

-  Personnel  support  organization/DRMD 

-  Force  protection 

-  Billeting/messing/local  transportation/medical 

-  Communications 

•  Set-up 

-  Testing/comm  checks/maintenance/contractor  support 

-  Calibration  of  instrumentation 

•  Redeployment/reconstitution/cleanup 


Getting  experimenters  (experiment  controllers,  data  collectors,  analysts,  etc.)  and 
participants  (units  and  augmentees  who  play  in  the  experiment)  to  the  various  events  is  a 
major  consideration.  USACOM  must  budget  and  POM  for  CJCS  transportation  dollars 
to  cover  all  planning,  training/spin-up  events  as  well  as  the  culminating  event.  Priority 
at  least  tis  high  as  CJCS-sponsored  exercises  must  be  accorded  joint  experimentation. 

Responsibility  for  bed  down  of  experimenters  and  participants  may  be  delegated  to 
Service  components,  but  USACOM  must  set  the  ground  rules,  manage  the  budget, 
ensure  safety,  security,  local  transportation,  billeting,  messing,  communications,  etc.  are 
sufficient  to  support  the  event. 

Sufficient  time  and  resources  must  be  allocated  for  set-up,  testing,  calibration,  etc.  to 
ensure  the  experiment  architectures  are  capable  of  supporting  the  event  and  that  a 
suitable  environment  is  provided  to  support  the  various  experiment  trials/initiatives. 

Experiment  sites  must  be  returned  to  a  condition  equal  to  or  better  than  what  existed 
before  the  experiment.  Budget  and  resources  must  be  sufficient  to  cover  cleanup  and 
redeployment  of  experimenters  and  participants. 
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Preparation  and  Conduct  of  Experiments 

Data  Collection  Planning 

•  Develop  data  collection  matrix  (requirements)  6  to  9 
months  prior  to  experiment  (Joint  effort  with  analysts) 

-  SME/data  collector  requirements  (tasking  #’s,  qualifications, 
iaydown,  training,  continuity) 

-  Instrumentation  (development,  laydown,  non-obtrusive) 

-  Interviews/surveys 

-  Collection  techniques  (automated,  manual) 

•  Consolidated  database  development  (Joint  effort  with 
analysts) 

-  Management 

•  Pilot  Test  -  End-to-End 

•  Data  collection  must  include  OPFOR 


It  is  important  to  ensure  that  the  data  collection  managers  be  brought  in  early  in  the 
process.  At  least  9  months  prior  to  the  experiment  the  analysts  should  provide  a  good 
idea  of  the  collection  requirements.  The  collection  requirement  will  dictate  the  number, 
type  and  qualifications  of  SMEs/data  collectors  and  the  quantity  and  type  of 
instrumentation  required.  Once  this  is  known  the  data  managers  can  begin  the  process 
of  acquiring  (tasking  for)  SMEs/data  collectors  and  obtaining  or  developing  the  required 
instrumentation.  This  process  can  be  very  time  consuming  depending  on  the 
requirements.  Automation  should  be  used  as  much  as  possible  to  support  manual  data 
collection.  Concurrently  the  data  managers  can  begin  developing  a  database  that  meets 
the  needs  of  both  the  collection  agency  and  analysis  agency.  Finally,  time  must  be 
allocated  for  a  pilot  test  or  an  end-to-end  test.  This  will  ensure  that  the  data  collection 
process  is  collecting  the  correct  data  and  that  the  analysts  are  satisfied  with  the  data. 
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Preparation  and  Conduct  of  Experiments 

Training 


•  Must  educate  senior  leaders  on  experiment 
issues  and  impact  of  changes 

•  All  need  to  be  aware  of  the  goals  and  objective  of 
the  experiment  and  role  of  others 

•  EXFOR  needs  TTP,  TSP,  NET  and  train-up 

•  OPFOR  needs  train-up 

•  Experimenters 

-  Need  specific  training 

-  Smart  book  and  hypotheses  card 


This  area  might  as  well  have  been  called  education  instead  of  training.  The  senior 
leadership  in  the  CINC  and  the  service  staffs  need  to  be  educated  on  the  scope  and  goals  of 
the  experiment.  This  can  be  accomplished  by  a  traveling  “road  show.”  One  goal  in  this 
education  process  is  right-sizing  their  expectations.  Another  goal  is  to  convince  them  that 
any  resources  they  are  “contributing”  are  being  used  in  a  great  cause  and  their  support  is 
required. 

Another  key  method  for  getting  everyone  involved  in  the  experiment  on  the  same  page  is  to 
develop  a  “smart  book”  and  “hypothesis  card.”  The  smart  book  is  specifically  designed  for 
SME’s  to  carry  and  it  identifies,  describes  and  explains  all  of  the 
hypotheses/issues/initiatives  in  the  experiment.  The  hypothesis  card  is  a  laminated 
business-size  card  that  succinctly  summarizes  the  goals  and  objectives  of  the  experiment. 
One  hundred  percent  of  the  participants  (to  include  data  collectors,  SMEs,  player  unit  and 
OPFOR)  should  be  required  to  carry  this  card. 
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Preparation  and  Conduct  of  Experiments 

Data  Collection  Execution 


•  Resources  and  requirements  must  be  identified 
early  in  order  to  manage  data  collection/ 
instrumentation/data  base  population  and  the 
quality  control  process 

•  Provide  quick-look  capability  (manual  and 
instrumented)  to  insure  getting  required  data  and 
to  support  daily  hot-wash 

•  Provide  analysts  access  to  experiment  and  data 
base 

•  Establish  a  data  review  process  to  validate 
subjective  data 

•  Ready  response  cell  of  SMEs/data  collectors 


Throughout  the  experiment  the  data  collection  manager  must  continue  to  work  with  the 
analysts  and  provide  data  as  soon  as  possible.  It  is  not  a  good  idea  to  wait  until  the 
experiment  is  completed  to  determine  if  adequate  data  was  collected.  Analysts  need  to 
ensure  that  the  data  is,  in  fact,  coming  in  as  the  experiment  is  in  progress.  This  allows 
for  the  possibility  to  adjust  data  collection  as  the  experiment  is  in  progress. 

Analysts  must  occasionally  get  away  from  their  computer  printouts  to  view  the 
experiment  first  hand  and  to  get  their  own  views  of  what  is  happening  in  order  to  better 
understand  the  data. 

Because  there  are  many  stakeholders  in  the  data,  a  formal  data  review  process  should  be 
established  to  authenticate  the  data  prior  to  releasing  the  database  for  final  analysis.  It  is 
especially  important  to  review  all  comment  data  for  relevance  and  veracity. 

Sometimes  collection  requirements  change  or  arise  during  execution.  The  data 
managers  should  have  a  handful  of  qualified  SMEs  to  fill  the  gaps  without  taking  away 
from  other  collection  requirements. 
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Preparation  and  Conduct  of  Experiments 

Control  of  the  Experiment 

•  If  experiment  is  in  conjunction  with  a  training 
exercise,  then  must  balance  training  objectives 
with  experiment  objectives 

•  Need  a  robust  control  ceil  that  includes  analyst 
representation 

•  Balance  free  play  and  MSEL  events 


Piggy-backing  experiments  on  service  exercises  is  probably  a  reality  for  some  time. 
There  will  be  continual  negotiations  to  insure  the  experimental  objects  do  not  get  lost. 
Need  to  establish  a  cell  to  continually  monitor  this  process.  Even  if  a  “pure”  experiment 
is  conducted,  there  still  needs  to  be  a  balance  between  achieving  specific  tasks  versus  the 
free  play  operational  realism  of  the  unit. 
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Preparation  and  Conduct  of  Experiments 
Protocol/Marketing/Visitor's 
Bureau _ 

•  Visitor  protocol 

-  Resource  visitor  bureau  —  pays  dividends  (. . .  but  non¬ 
interference) 

•  standard  briefing/tours  (to  include  into  experimental  unit), 
opportunity  to  right-size  expectations,  send  out  invitations,  VIP 
treatment 

-  Need  high-level  support  to  enforce  “no  additional  data 
collectors  /  observers” 

-  Consider  virtual  visitor  bureau 


Experiments  are  expensive  propositions.  Senior  leaders  (service,  joint  and  DoD)  as  well 
as  Congress  want  to  ensure  the  money  is  well  spent.  We  cannot  ignore  the  political 
reality.  ...they  will  want  to  see  the  experiment  firsthand  to  validate  that  the  money  was 
well  spent.  Therefore,  thorough  planning  for  visitors  is  critical  to  the  success  of  the 
experiment.  The  difficulty  lies  in  the  creative  tension  between  visitor  support 
(especially  as  the  rank  of  the  visitor  increases)  and  not  interfering  with  the  experiment 
itself.  Collectively,  this  could  be  called  “experimentation  marketing.”  Resourcing  a 
force  to  manage  this  “campaign”  is  critical  to  its  success.  Invitations  for  very  senior 
leaders  is  a  long-lead  time  item  and  will  be  subject  to  numerous  perturbations  as 
scheduling  conflicts  appear.  Determining  what  should  be  on  the  visit  itinerary, 
appropriate  protocol  to  be  rendered  etc.,  take  time,  effort  and  especially  attention  to 
detail.  Ensuring  that  the  invitation  list  (and  more  importantly  those  that  “invite” 
themselves)  is  manageable  also  requires  senior  leader  oversight.  Expectation 
management  is  another  critical  component. 


58 


Preparation  and  Conduct  of  Experiments 
Maintaining  Experimentation 
infrastructure _ 

•  Care  and  feeding  of  experimenters 

-  Shift  schedule  coordination 

-  Transportation  needs 

-  Communication  needs  (ceii  phones,  fax) 

-  Lodging/messing/medical 

•  Computer/communication  networks 

-  On  site  maintenance  support 

-  Dedicated  networks 


Once  the  experiment  begins  it  is  critical  that  the  different  elements  comprising  the 
experimental  environment  receive  proper  attention  by  the  operations  group.  The  various 
aspects  of  the  infrastructure  require  “care  and  feeding”  to  ensure  that  the  live  experiment  is 
properly  supported  and  completed  on  schedule.  The  major  parts  of  the  infrastructure  consist 
of  the  personnel  (experimenters)  conducting  the  experiment  and  the  equipment  needed  to 
support  the  experiment. 

Basic  needs  of  the  experimenters  during  the  experiment  include  lodging,  food, 
transportation,  medical,  and  communication.  The  numbers  of  personnel  conducting  the 
experiment  and  the  duration  of  the  experiment  are  key  drivers  for  these  support  areas. 
Another  issue  is  the  experimenter  shift  schedule.  If  an  insufficient  number  of  experimenters 
is  used,  the  shift  schedule  can  become  infeasible  or  so  stressful  it  will  adversely  impact  the 
overall  conduct.  Health  of  the  experimenter  group  needs  to  be  maintained  to  prevent  any 
possible  shutdown  of  different  parts  of  the  experiment.  Consider  requirement  estimates  for 
“supernumeraries”  for  key  experimental  staff  an  adjustment  factor  for  members  who  may 
become  ill  to  provide  for  alternate  or  “back-up.” 

Equipment  to  support  the  conduct  of  the  experiment  includes  such  items  as  the  computer 
hardware  and  software,  communications  network,  transportation,  repair  parts  and  power 
generation  used  by  the  experimenter  staff.  Maintaining  computers,  the  related  Local  Area 
Network  (LAN),  and  the  distributed  network  is  critical.  An  on-site  dedicated  technical 
maintenance  support  center  to  provide  responsive  computer  support  should  be  considered. 

If  possible,  a  dedicated  communications  network  should  be  used  for  the  experiment.  The 
transportation  and  power  generation  equipment  requires  a  maintenance  staff  to  do  daily 
checks.  The  maintenance  staff  needs  an  adequate  level  of  spares  to  repair  equipment. 
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Preparation  and  Conduct  of  Experiments 
Hot  Wash  Briefing  Requirements 

•  Don ’t  fight  the  appetite  for  immediate  analytic 
feedback 

•  Establish  a  process  for  generating  daily  or  cyclic 
analytic  feedback 

•  Use  the  process  to  capture  analytic  insights  while 
fresh 

•  Train  analysts  to  do  analysis  of  incomplete  data 

•  Select  credible  briefer  (military  analysts) 


The  experimentation  community  must  realize  that  there  will  be  a  natural  requirement  on 
the  part  of  senior  leadership  to  have  some  immediate  and  cyclic  analytic  feedback 
during  the  course  of  the  experiment/exercise.  This  is  mainly  caused  by  the  need  to  show 
some  immediate  results  to  those  who  provided  the  large  investment  of  time  and  dollars 
in  the  experiment.  The  experimentation  community  should  not  try  to  resist  this 
tendency,  but  rather  should  use  it  as  an  opportunity  to  conduct  some  level  of  analysis 
during  the  exercise  while  the  events  are  still  fresh  in  their  minds. 

A  process  for  conducting  analysis,  which  leads  to  this  feedback,  must  be  established. 
Feedback  should  be  expected  on  a  daily  basis,  or  perhaps  on  some  other  cyclic  basis 
(e.g.  at  the  end  of  each  phase  of  a  scenario). 

The  analysts  must  be  trained  to  conduct  analysis  on  incomplete  data  in  order  to  produce 
this  immediate  analytic  feedback.  This  will  be  uncomfortable  to  most  analysts  who 
prefer  to  have  the  complete  set  of  data  from  the  experiment. 

These  analytic  results  should  be  presented  by  an  analyst  who  is  credible  in  the  eyes  of 
the  audience  (probably  the  senior  leadership).  In  the  case  of  major  experiments,  this  will 
most  likely  be  a  military  analyst. 
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Preparation  and  Conduct  of  Experiments 
Analysis  and  Reporting 


•  Apply  the  Model-Exercise-Model  (MEM)  approach,  if 
possible 

•  Establish  a  process  for  conducting  analysis 
throughout  the  experimentation  process 

•  Focus  analysis  on  potential  capabilities,  not  specific 
systems 

•  Conduct  post-event  analysis  workshop(s). 
Experiment  is  not  concluded  until  MEM  is  completed. 

•  Produce  one  Integrated  report  preceded  by  a  short, 
hard-hitting  Initial  Insight  Memo 


When  possible,  the  Model-Exercise-Model  (MEM)  analysis  approach  should  be  used  for 
experimentation  so  that  the  complete  set  of  analytic  issues  are  addressed  by  the 
appropriate  tool,  whether  that  be  a  model,  or  the  live  exercise  event.  This  requires  the 
establishment  of  an  analytic  process  which  will  enable  initial  analytic  conclusions  to  be 
drawn  using  incomplete  data  from  the  exercise. 

Because  an  experiment  is  not  typically  used  to  directly  support  the  material  acquisition 
process,  it  should  focus  on  the  identification  and  evaluation  of  potential  capabilities 
demonstrated  within  the  experiment,  not  on  the  evaluation  of  specific  systems. 

To  facilitate  the  analysis,  given  that  much  of  it  is  being  conducted  by  groups  of  analysts 
who  are  geographically  dispersed  after  the  experimental  event,  post-event  analysis 
workshops  should  be  conducted.  The  focus  of  these  efforts  is  to  produce  two  products. 
The  first  product,  an  Initial  Insights  Memorandum,  is  a  short  accounting  of  the  major 
insights  drawn  from  the  experiment,  even  if  produced  from  incomplete  data.  This 
product  is  intended  to  satisfy  the  immediate  appetite  for  analytic  results  by  the  senior 
leadership. 

The  second  product  is  the  final  report  for  the  experiment.  This  report  must  be  a  single 
document  which  contains  all  analytic  results  of  the  experiment,  regardless  of  who 
developed  them.  A  single  report  is  necessary  to  ensure  that  contradictory  results  from 
multiple  reports  are  not  released  to  the  consumers. 
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MILITARY  OPERATIONS  RESEARCH  SOCIETY 


Joint  Experimentation 
Workshop 

Modeiing  and  Simuiation  WG 
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Agenda 


Modeling  and  Experimentation 


•  Participants 

•  Working  Group  Tasks 

•  Working  Group  Results 

•  Key  Issues  and  Questions 

•  Actionable  Recommendations 
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The  members  of  the  working  group  (and  subsequent  sub- working  groups)  drew  heavily 
from  active  duty  military,  academic  and  private  sector  experts  representing  a  broad 
cross-section  of  the  Modeling  and  Simulation  (M&S)  community.  Direct  experience  in 
the  planning,  staging  and  development  of  Joint  Experiments  (JE)  provided  the  expertise 
needed  to  scope  and  bound  major  problems. 
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Modeling  and  Experimentation 

Working  Group  Tasks 

•  Address  Modeling  and  Simulation  (M&S) 

aspects  of  terms  of  reference  within  a  Model- 

Experiment-Model  framework 

-  Identify  methodologies  and  tools 

-  Identify  critical  M&S  issues  in  concept  development/ 
planning,  experiment  conduct,  and  exploitation 

-  Identify  ways  to  federate  and  utilize  existing  models 
Evaluate  characteristics  of  available  models  to  assess 
their  applicability  to  experiments 

-  Identify  requirements  for  future  M&S  capabilities 

-  Establish  practical  limits  to  M&S  in  large  scale 
experiments 


The  working  group  focused  on  those  aspects  of  the  conference  Terms  of  Reference 
(TOR)  most  pertinent  to  M&S  within  the  model-experiment-model  notional  framework. 
The  group  applied  the  following  three  criteria  to  select  issues  to  explore  and  report: 

•  Joint  Application  -  the  issue  is  important  for  joint  experimentation  rather  than 
single  service  application. 

•  High  impact  -  the  issue  has  a  significant  impact  on  joint  experimentation. 

•  Consensus  -  the  conclusion  reflects  the  consensus  of  the  working  group  (and  in 
some  cases  the  sub-working  group). 

To  develop  the  material  for  this  report,  the  working  group  divided  into  four  smaller 
groups  that  discussed  the  following: 

1)  Use  of  M&S  for  concept  development. 

2)  Experiment  design. 

3)  The  conduct  of  the  experiment. 

4)  Exploitation  of  experimental  results. 

Because  the  working  group  felt  that  they  lacked  sufficient  experience  and  exposure  to 
experimental  campaigns,  the  group  chose  not  to  form  a  fifth  subgroup  to  address  use  of 
M&S  to  support  experimental  campaigns.  However,  each  of  the  subgroups  did  address 
experiment  campaign  issues  as  part  of  their  work. 
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Modeling  and  Experimentation 

Working  Group  Results  (1  of  5) 


•  Methodologies  and  tools 


-  For  Exploiting  experiment 

•  For  identifying  cause  and  effects  chain 

•  For  conducting  excursions 

•  For  conducting  sensitivity  analysis 

•  For  determining  parameter  values 

•  For  feed  forward 


M&S  can  support  joint  experimentation  by  helping  experimenters  to  represent  the  future  by  reducing  costs 
and  by  increasing/enabling  range  safety.  Models  and  simulations  can  provide  continuous  support  throughout 
an  experimental  campaign.  They  can  help  establish  concept  baselines  for  future  explorations.  By  requiring 
the  knoAvledge  to  be  represented  fully  and  explicitly,  they  can  improve  knowledge  acquisition.  The  types  of 
tools  used  changed  during  the  experimental  campaign.  The  tools  used  later  in  the  campaign  are  generally 
more  refined  and  complex  than  those  used  earlier  in  the  campaign. 

The  exploitation  group  examined  methodologies  and  tools  in  depth.  Their  analyses  recommended  methods 
for  reconstructing  the  experiment,  examining  unplanned  opportunities  resulting  from  the  experiment, 
identifying  cause  and  effects  chain,  conducting  excursions,  conducting  sensitivity  analysis,  feeding  forward 
results  into  subsequent  events  in  the  experimental  campaign  and  summarizing  and  disseminating  results. 

M&S  can  support  joint  experimentation  by  helping  experimenters  to: 

•  Document  the  growing  understanding  of  the  conceptual  doctrine,  organization  or  technology  being 
examined. 

•  Understand  how  the  concept  improves  achievement  of  specified  objectives. 

•  Refine  the  completeness  and  coherence  of  the  concept. 

•  Assess  the  feasibility  of  a  concept. 

•  Illustrate  the  dynamics  of  variables,  identify  causal  relationships  and  demonstrate  the  implications 
of  these  relationships. 

•  Refine  the  scenario  space  (e.g.,  it  helps  experimenters  examine  where  a  concept  has  the  greatest 
impact,  and/or  where  it’s  contribution  is  most  limited). 

•  Examine  the  impact  of  specific  threats,  countermeasures  and  performance  required  to  overcome 
countermeasures. 

The  working  group  decided  not  to  discuss  specific  models  by  name  but  instead  to  focus  on  model 
characteristics  important  to  different  applications  of  M&S.  This  decision  facilitated  achievement  of 
workshop  objectives  by  removing  the  potentially  divisive  comparisons  of  model  capabilities. 
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Modeling  and  Experimentation 

Working  Group  Results  (2  of  5) 

•  Practical  limits  of  M&S 

-  Multi-level  security 

-  Unable  to  model  relationships  that  are  not  understood  in  the  real  world 

-  Ability  to  rapidly  represent  new  concepts 

-  Inability  to  build  a  predictively  accurate  mode!  of  a  chaotic  process 

•  Deficiencies  of  current  M&S 

-  Various  service  models  lack  credibility  among  sister  services  of  joint  domain 

-  Lack  of  High  Level  Architecture  (HLA)  compliance,  interfaces  to  real  world 
systems/cross-level  representation  and  semantic  commonality/logical 
consistency  among  federates 

-  Models  do  not  adequately  reflect  impact  of  C4ISR,  10,  human  behavior 
(individual  and  group),  logistics,  WMD  and  geophysical  environment  on 
weapon  system  and  sensor  performance 

-  Inadequate  environmental  representations  (common  representations  - 
databases,  dynamic  effects,  Interactions  of  terrain,  ocean,  DIED  limits 

-  Ability  to  represent  Operations  Other  Than  War  (OOTW)  and  Military 
Operations  in  Urban  Terrain  (MOUT)  largely  non-existent 

_ -  Lack  of  adequately  verified  and  validated  tools _ 

The  working  group  distinguished  between  “Practical  Limits”  and  “Deficiencies.” 
"Practical  limits"  addressed  limits  that  are  fundamental  to  the  nature  of  M&S  and 
(arguably)  cannot  be  resolved.  "Deficiencies"  are  current  shortfalls  that  the  working 
group  deemed  correctable. 

One  practical  limit  of  particular  interest  was  the  inability  to  build  predictively  accurate 
models  of  chaotic  processes.  Building  predictive  models  of  many  non-chaotic  processes, 
including  adversary  responses,  remains  a  very  difficult  (if  not  impossible)  problem 
whose  difficulty  may  be  neither  fully  understood  nor  appreciated  by  many  consumers  of 
M&S. 

The  following  elaborates  on  several  of  the  current  deficiencies  in  M&S: 

•  Lack  of  model  credibility  among  sister  services  of  joint  domain.  Workshop 
participants  generally  felt  that  each  service  lacks  confidence  in  models  developed 
by  sister  services.  Each  service  believes  that  sister  services  do  not  represent  the 
issues  adequately  for  their  needs.  For  example,  the  Air  Force  may  feel  that  the 
Army’s  ground  war  models  do  not  represent  the  air  war  as  required  to  meet  Air 
Force  needs.  Similarly,  the  Army  may  feel  that  Air  Force  air  combat  models  do 
not  replicate  ground  combat  as  needed.  The  failure  of  overlapping  regions  of 
each  model  to  agree  supports  this  view.  In  addition,  trying  to  analyze  joint 
experimental  results  using  service-specific  simulations  impedes  the  ability  to 
gain  insights  into  the  synergism  of  joint  operations. 

•  Inability  to  model  impact  of  communications  and  human  behavior.  Though 
models  can  adequately  represent  the  dissemination  of  information,  it  is  difficult 
to  show  how  people  process  and  use  the  information  to  make  decisions. 
Therefore,  the  assessment  of  the  contribution  of  information  technologies  to 
performance  and  effectiveness  is  often  difficult  and  incomplete. 
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Modeling  and  Experimentation 

Working  Group  Results  (2  of  5  cont) 


•  Difficulty  in  representing  new  concepts  using  existing  models  or  data.  Futuristic  systems 
or  concepts,  that  by  their  very  nature  are  not  well  defined,  require  significant  changes  in 
the  assumptions  of  the  existing  model.  Required  model  modifications  increase  costs  and 
lead  time  to  prepare  models. 

•  Lack  of  models  that  represent  OOTW  and  MOUT.  Most  current  models  are  tailored  for 
mid-  to  high-intensity  conflict.  Because  emerging  missions  such  as  operations  other  than 
war  or  military  operations  in  urban  terrain  are  complex  and  not  well  understood,  changing 
existing  models  or  developing  new  ones  able  to  address  this  environment  will  take  time 
and  significant  resources.  The  next  generation  of  models  and  simulations  must  address 
this  changing  world  situation. 

•  Lack  of  model  output  able  to  support  new  MOEs.  MOPs  for  emerging  missions.  Most 
current  models,  built  to  examine  firepower  and  maneuver  warfare,  do  not  produce 
products  that  relate  easily  to  the  Measures  Of  Effectiveness  (MOEs)  and  Measures  Of 
Performance  (MOPs)  appropriate  to  new  emerging  missions.  New  concepts,  such  as 
information  operations  and  network  centric  warfare  will  require  new  analytical  techniques 
to  produce  the  needed  MOEs  and  MOPs.  For  example,  there  is  a  need  for  models  that 
show  the  effects  of  psychological  operations,  non- lethal  weapons  and  suppressive  effects. 

Other  deficiencies  of  current  M&S,  in  addition  to  those  in  the  viewgraph,  include: 

•  The  lack  for  scenario  stability  such  as  would  be  provided  by  a  standard 
illustrative  DPG  scenario. 

•  The  lack  of  common  approaches  to  standard  scenario  elements,  including  lack  of 
a  common  approach  to  missile  endgame. 

•  The  lack  of  balance  within  models  between  size,  force  and  level  of 
representation. 

•  The  need  for  interfaces  to  real  world  C4I  systems  and  coalition  models. 

•  A  lack  of  awareness  of  the  dangers  involved  with  examining  new  concepts  with 
models  developed  for  previous  studies  and  analyses. 
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Modeling  and  Experimentation 

Working  Group  Results  (3  of  5) 

•  Federation  and  use  of  models 

-  Integration  requirements  and  ways  to  meet 
them 

•  Guidance  provided  by  Defense  Modeling  and 
Simulation  Office  (DMSO)  Federation  Execution 
and  Development  Process 

•  Integrate  with  concept 

•  Integrate  with  system  of  systems  (human,  C4ISR, 
federates,  training  plan,  data  collection 
(instrumentation  and  analysis).  Verification, 
Validation  and  Accreditation  (VV&A) 

•  Integrate  with  warfighter  confidence 


Use  of  the  Defense  Modeling  and  Simulation  Office  Federation  Execution  and 
Development  Process  (FEDEP)  is  highly  recommended,  particularly  the  checklist  for 
federations  designed  for  experiments,  related  security  and  verification,  validation  and 
accreditation  overlays. 

Before  the  experiment  is  conducted,  the  simulation  must  be  modified  to  best  mirror  the 
concept  being  evaluated.  This  may  be  accomplished  by  federating  models  that  properly 
address  different  aspects  of  the  concept. 

Consideration  for  integration  of  supporting  models  and  simulation  tools  must  occur 
throughout  the  experimental  process  (starting  with  concept  development  and  ending  with 
exploitation).  This  integration  should  occur  continuously  and  iteratively,  as  in  spiral 
development.  For  example,  as  the  concept  is  modified,  so  must  the  supporting  models  and 
simulation  tools. 

Within  this  integration  effort,  a  system  of  systems  approach  must  be  used.  This  approach 
should  consider  human  participants/observers/controllers,  C4ISR  systems,  federates, 
training  plans,  data  collection  (instramentation)  and  analysis  resources  and  V&VA 
resources.  To  support  accreditation  and  “buy-in,”  the  warfighter  must  establish  confidence 
in  M&S.  The  accreditation  process  must  identify  and  address  all  stakeholders'  issues.  If 
accreditation  cannot  be  achieved,  then  the  experimental  results  will  not  be  accepted  by  the 
warfighter  (user/customer). 
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Modeling  and  Experimentation 

Working  Group  Results  (4  of  5) 

•  Model  selection  criteria 

-  Operational  (ability  to  support  functional  description  of  the  experiment, 
level  of  aggregation,  terrain  and  environment,  scope,  resolution) 

-  Has  a  representation  that  is  both  necessary  and  sufficient  for  concept 
being  explored  (avoid  unnecessary  detail) 

-  Verification  and  validation  status  and  warfighter  confidence 

-  Technical  (number  of  units,  time-management,  run  speed,  interoperable 
with  other  simulations  and  systems,  reuse,  security  level) 

-  Resource  availability  (database  availability,development  time,  portability, 
support  staff,  facilities,  equipment) 

-  Milestone  and  timeline  constraints 

-  Human  involvement,  C4ISR  (real/synthetic  representation) 

-  Potential  for  reuse  in  overall  experimentation  campaign 

-  Modification  requirements  and  risk 

-  Adequacy  of  Configuration  Management  (CM)and  Life  Cycle  Maintenance 
process  (LCM) 


There  are  many  criteria  for  selecting  the  appropriate  modeling  and  simulation  tools. 

Selection  criteria  include: 

•  Experiment  objective.  Ability  to  support  the  experiment  objective  is  the  most 
important  consideration.  It  includes  the  ability  of  the  M&S  tools  to  produce  the 
data  needed  by  the  evaluation  metrics;  and  to  address  the  appropriate  levels  of 
warfare,  current  or  future  capabilities,  and  systems  and  doctrine.  The  model  must 
have  the  level  of  resolution  required  to  capture  the  desired  representations.  The 
model  should  fit  the  problem,  not  the  problem  to  the  model. 

•  Representation  of  needed  detail.  Complexity  and  detail  should  be  introduced  when 
needed,  and  avoided  when  not  needed.  For  example,  experiments  that  emphasize 
environmental  representation  effects  (e.g.,  IR  and  MMW  reflective  properties  of 
vegetation)  require  models  and  simulations  that  can  capture  that  level  of  detail. 

•  Verification  and  Validation.  Accreditation  is  needed  both  for  general  and  particular 
uses  of  the  model.  Verified  and  Validated  (V&V)  tools  should  be  used  as  often  as 
possible.  If  a  non-V&V  tool  must  be  used,  then  time  and  resources  must  be 
allocated  to  achieve  V&V.  If  not,  experimental  results  may  be  suspect.  Note  that 
V&V  is  less  important  early  in  the  experimental  campaign. 

•  Ease  of  integration.  In  live  and  virtual  simulations,  the  integration  of  real-world 
C4ISR  systems  is  critical.  C4ISR-simulation  interfaces  have  usually  been 
developed  as  point-to-point  solutions  since  modular  interfaces  are  not  readily 
available.  One  should  select  models  and  simulations  that  already  have  existing 
C4ISR-simulation  interfaces  or  support  development  of  an  interface. 
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Modeling  and  Experimentation 
Working  Group  Results  (4  of  5  cont.) 


•  The  availability  of  resources.  One  must  first  leverage  what  is  available  for  the  live 
portion  of  the  experiment  and  then  fill  in  gaps  with  constructive  and  virtual 
simulations.  Other  resource  considerations  are  personnel  (DoD  and  contractor) 
available  to  support  the  experiment,  facilities  and  equipment  (including  hardware, 
software,  databases)  and  other  logistical  considerations. 

•  Milestone  constraints.  These  include  the  time  period  designated  for  the  experiment 
(within  the  experimental  campaign  plan).  The  set-up  time  and/or  modification  time 
required  for  the  employment  of  a  particular  tool  must  fit  into  the  allotted  schedule. 

•  Roles  of  people.  Human  involvement  requires  use  of  virtual  or  live  simulations. 
Synthetic  representations  of  humans  and  their  decision  processes  are  not  mature 
enough  to  be  credible  within  constructive  simulations. 

•  Support  to  long  term  interoperability  and  reuse.  During  the  earlier  phases  of  the 
experimental  campaign,  model  and  simulation  selections  should  consider  the 
candidate  model’s  ability  to  support  long  term  interoperability  and  reuse  for  follow-on 
experiments  within  the  experimental  campaign  plan.  Models  should  be  HLA 
compliant.  Data  driven  models  that  are  easy  to  change,  (particularly  for  unit  behavior 
and  C2  relationships)  are  recommended. 

•  Ability  to  support  the  data  collection  plan.  Proper  instrumentation  facilitates  data 
collection.  It  does  not,  however,  address  all  data  collection  requirements,  especially 
data  collected  from  SME  controllers  and  qualitative  data. 

•  Standard  issues  of  Multiple  Levels  of  Security  fMLvS).  The  collaborative,  distributed 
nature  of  these  experiments  compounds  the  MLS  problem. 
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Modeling  and  Experimentation 

Working  Group  Results  (5  of  5) 

•  Requirements  for  future  M&S  capabilities 

-  W&A  process  for  joint  models  and  joint  use  of  Service 
models 

-  Broader  use  of  class  accreditation  where  appropriate 

-  Data-driven  models  that  are  easy  to  change,  particularly  for 
unit  behavior  and  C2  analyses 

~  Approved  joint  future  scenarios  and  an  agreed  to  set  of 
baseline  standard  future  scenarios 

-  Ability  to  collaboratively  plan  and  execute  actions  from 
geographically  distributed  locations 

-  Direct  observable  results  (real-time  feed  back) 

-  Predictively  accurate  large  scale  models 

-  Input  and  output  linkage  to  real  systems 

-  HLA  compliance 


Models  and  simulations  have  many  shortcomings  in  representing  the  real  world.  Perhaps  the  most 
significant  current  limitation  concerns  the  availability  of  predictively  accurate  large  scale  models.  For 
most  experimentation  purposes,  models  must  be  sufficiently  accurate  so  that  the  conclusions  about  the 
concepts  being  evaluated  are  valid.  Areas  where  sufficient  predictive  accuracy  is  not  currently 
available  include  representations  of  human  behavior,  logistics,  some  environmental  effects  within 
joint  scenarios. 

•  Human  behavior.  The  representation  of  human  behavior  and  of  human  cognitive  processes 
is  recognized  as  a  major  issue  within  the  DoD  M&S  community.  This  shortfall  impairs  our 
ability  to  model  the  impact  of  advanced  C4ISR  systems  and  information  operations  on 
human  decision  processes,  and  of  human  decision  processes  on  mission  outcome.  The 
working  group  believes  that  near-term  solutions  for  this  shortfall  are  unlikely. 

•  Logistics.  The  effect  of  logistics  (especially  shortages)  on  operational  decision-making  is 
also  not  properly  modeled. 

•  Environmental  models.  Environmental  models  remain  inadequate  to  support  many  aspects  of 
joint  experimentation.  Such  modes  either  do  not  exist  or  are  too  computationally  intensive  to 
support  evaluations  of  weapon  system  and  sensor  performance  within  simulations.  It  has 
also  been  difficult  to  identify  V&V’d  models  of  the  dissipation  and  effects  of  WMD. 
Computational  fluid  dynamic  approaches  are  still  too  inunature  and  processing-intensive  to 
provide  acceptable  models,  especially  in  built-up  areas.  Adequate  environmental 
representations  should  include  common  database  representations,  dynamic  effects  and 
interactions  (holes  are  left  by  bombs,  rain  causes  mud  and  swollen  rivers,  waves  wash  away 
beaches,  tides  change,  etc.),  and  greater  accessibility  of  higher  resolution  databases. 

•  Approved  joint  future  scenarios  fscenario  stability  V  It  is  important  to  understanding  better 
the  relative  merit  of  having  one  joint  scenario  across  multiple  experimental  events  or  of  tying 
service  specific  scenarios  to  each  other.  We  need  to  understand  better  the  requirements  for 
seamless  scenarios. 
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Modeling  and  Experimentation 

Key  Issues  and  Questions 


•  Lack  of  a  joint  validation  process 

•  Acceptability  of  service-unique  tools  in  a  joint 
environment 

•  Limited  state-of-the-art  in  ability  to 

-  Constructively  model  the  human  decision  process 

-  Accommodate  the  impact  of  "so/T  factors 

•  Limited  ability  to  provide  models  of  the  experimental 
scenario 

-  Limits  of  knowledge  of  phenomena 

-  Lack  of  available  M&S  that  treats  features  of  new 
concepts 

•  JE  requires  a  full  spectrum  of  federated  components 

•  Developing  Warfighter  confidence  in  M&S  and  the  process 


This  slide  reiterates  the  issues  that  the  working  group  felt  were  most  important. 

Annotations  elaborating  on  these  items  accompanied  the  viewgraphs  where  the  issues  were 
first  presented.  Summaries  of  the  annotations  are: 

•  Lack  of  joint  validation  process.  This  lack  impacts  the  inter-service  acceptability  of 
tools  developed  by  one  service  principally  for  their  own  use. 

•  Acceptability  of  service-unique  tools  in  a  joint  environment.  Lack  of  cross-service 
credibility  stems  from  differences  in  types  of  issues  that  are  modeled  and  objectives 
of  the  model.  Often,  the  overlapping  regions  of  each  model  tend  to  disagree. 

•  Limited  ability  to  constmctivelv  model  the  human  decision  process.  It  is  difficult  to 
understand,  and  hence  model,  how  people  use  available  information  to  make 
decisions.  Consequently,  it  is  difficult  to  assess  the  contribution  of  information 
technologies  to  performance.  Analyses  of  information  effectiveness  are  difficult 
and  incomplete. 

•  Ability  to  provide  models  of  the  experimental  scenario.  Most  current  models  are 
tailored  for  mid-  to  high-intensity  conflict.  There  is  limited  ability  to  represent 
OOTW  and  MOUT.  Addressing  this  limitation  will  likely  require  significant 
resources,  both  because  of  limits  of  knowledge  of  phenomena  and  because  of  lack 
of  available  M&S  that  treats  features  of  new  concepts. 

•  M&S  that  plays  in  a  full  spectrum  of  JEs  requires  federated  components.  HLA 
federates  are  not  readily  available.  Creating  new  federations  are  difficult.  Needed 
federations  are  not  usually  available  “off  the  shelf.”  Bringing  legacy  models  into 
HLA  compliance  is  often  impractical. 

•  Developing  warfighter  confidence  in  M&S  and  processes.  Because  it  is  difficult  to 
understand  the  assumptions  and  processes  employed  by  models,  warfighters  cannot 
be  confident  that  model  results  are  valid  for  the  purposes  of  any  particular 
experiment  objective. 
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Modeling  and  Experimentation 
Actionable  Recommendations  (1  of  2) 


•  DMSO  sponsor  “tiger  team”  to  identify  process  to  better 
integrate  modeling  and  simulation  tools  into  the  joint 
experimentation  process 

-  Would  support  on-going  simulation  based  acquisition  effort 

•  DMSO  Environmental  Technical  Working  Group  examine  and 
report  on  the  standard  characteristics  necessary  for  uniformity 
of  a  global  terrain  database 

•  J-9  identify  a  sponsor  to  establish  guidelines  for  using  training 
and  analysis  tools  in  experimentation 

•  Consider  live  simulations  for  OOTW  experimentation 

-  Evaluate  current  OOTW  tools 

-  Develop  requirements  for  new  OOTW  models  and  simulations;  especially 
models  of  non-lethal  weapons 

•  DMSO  V&V  technical  working  group  establish  formal  VV&A 

process  for  joint  M&S  and  joint  use  of  service  M&S _ 


•  DMSO  sponsor  ‘Tiger  Team.”  Since  joint  experimentation  is  so  new,  the  working  group 
felt  that  a  short-term  team  (sponsored  by  DMSO)  should  be  organized  to  better  understand 
the  process  of  integrating  M&S  into  joint  experimentation. 

•  Need  for  uniformity  in  terrain  database.  The  demand  for  terrain  at  various  locations 
around  the  world  requires  a  standardized  approach  to  ensuring  a  consistent/seamless 
product  that  can  be  used  by  joint  and  service-specific  experiments. 

•  Training  guidelines.  Training  tools  emphasize  realism  for  human  operators  and  sometimes 
lack  physics-based  details.  Analysis  tools  are  robust  in  physics-based  representations,  but 
often  lack  realism.  A  balance  must  be  struck  between  the  two  if  detail  is  required  and 
realism  is  desired. 
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Modeling  and  Experimentation 
Actionable  Recommendations  (2  of  2) 


•  Adopt  model-experiment-model  approach 

•  Develop  common  joint  semantics 

•  Ensure  that  concept  and  design  phases  are  continuous  and 
iterative 

•  Appropriately  use  humans  and  live  systems  for  assessing 
impact  of  information  operations  and  C2  systems 

-  Capture  decision  making  data  to  support  development  of  human 
behavior  representations 

•  Participate  in  future  M&S  development  and  enhancements 
(e,g,,  JWARS,  Combat  XXI)  to  ensure  their  suitability  to 
support  JE 

•  Use  a  decision  matrix  to  select  best  M&S  tools  for 
supporting  each  experiment 


•  Common  joint  semantics^  An  example  of  lack  of  common  semantics  is  in  use  of  the  term 
“exercise.”  The  Army  says  it  is  an  exercise,  not  an  experiment.  The  Air  Force  says  the 
opposite. 

•  OOTW  for  live  simulations.  OOTW  are  typically  not  well  represented  in  current  M&S 
tools  (which  are  based  on  Cold  War  scenarios.)  Though  some  OOTW  models  exist; 
requirements  for  these  mission  areas  should  be  developed  to  meet  future  needs.  For 
example,  models  require  representations  of  non-lethal  weapons, 

•  Use  of  humans  and  live  systems.  Whenever  an  experiment  examines  decision  making, 
command  and  control,  and  the  processing  of  information,  then  humans  and  actual  or 
credible  representations  (for  future)  of  C4ISR  systems  should  be  employed.  This 
requirement  stems  from  the  immature  state  of  art  for  representing  these  processes.  When 
available,  data  on  human  decision  making  and  behavior  should  be  collected  and  used  for 
future  constructive  representations  of  human  behavior  and  cognitive  processes. 

•  Use  of  decision  matrix.  A  decision  matrix  that  uses  selection  criteria,  as  described  in 
viewgraph  9,  is  strongly  recommended  as  an  aid  for  determining  the  best  models  and/or 
simulations  for  an  experiment.  Criteria  in  addition  to  those  described  in  this  report,  would 
address  other  factors  important  to  model  selection. 
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Assessment  and  Utilization  of  Results  Working  Group 

Working  Group  Participants  lUl^SiSll 

Jacqueline  Henningsen,FS,  WG  Chair 

HQ  USAF  AXOC 

Priscilla  Glasow,  WG  CoChair 

MITRE 

Linda  Weber,  WG  Co-Chair 

MITRE,  USACOMJ97 

J±.  Adkinson 

MCWL  (EDO,  TSA) 

John  Baird 

Raytheon  Systems 

Robert  Fleming 

Camber  Corp. 

Christine  Fossett,  FS 

GAO 

Frank  Gross 

CAI,  USACOMJ97 

Dean  Hartley,  III 

Data  Systems  R&D  Prog, 

Floyd  Kennedy,  Jr. 

Submarine  Forces  Atlantic 

LTC  Michael  Kwan 

TRADOC  Analysis  Center 

Ronald  Magee 

TRADOC  Analysis  Center 

Mark  Murray 

HQ  TRADOC  (Utilization) 

CAPTJD  Oliver 

USACOM  J97/J98 

Randall  Schmidt 

BETAC,  AFEO 

Richard  Simpson  (NATO) 

Allied  Command  Atlantic 

Donald  Theune 

Virtual  Technology  Corp. 

LTC  Fletcher  Thornton 

Joint  Staff  J8 

RADM  (ret)  Gary  Wheatley 

Evidence  Based  Research 

Leland  Joe 

Synthesis,  RAND 

Jerry  Kotchka 

Synthesis,  MORS  Past  President 

Chuck  Volmer 

VII  Inc 

Lt  Gen  (ret)  Steve  Croker,  USAF 

EFX  Senior  Mentor 

Working  group  members  and  their  affiliations  are  shown  on  this  slide. 
Composition  of  the  Group  (does  not  include  synthesis  participation): 
10  Govemment/Military 
2FFRDC 
7  Contractors 


Functional  Composition: 

2  United  States  Air  Force 

3  United  States  Army 

1  United  States  Marine  Corps 
1  Navy/Joint 

5  Joint  (1  JS,  3  USACOM,  1  Jt  Doctrine) 
1  Office  Secretary  of  Defense 
1  USG  (GAO) 

1  DOE 
1  NATO 

3  Other  (2  M&S,  1  AIAA) 
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Assessment  and  Utilization  of  Results  Working  Group 

Working  Group  Objectives 


•  Identify  the  critical  issues  in  assessing  and  utilizing 
the  results  of  experiments 

•  Develop  pragmatic  guidelines  for  conducting  valid 
experiments  whose  results  can  be  shared,  while 
supporting  the  needs  of  the  operational  community 


The  objectives  of  this  working  group  focused  on  identifying  issues  and  promising 
approaches  for  assessment  and  utilization  of  experimental  results.  We  also  explored  the 
relationship  between  validation  of  M&S  and  validation  of  experiments. 
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Assessment  and  Utilization  of  Results  Working  Group 


Working  Group  Tasks 


•  Identify  the  issues  that  are  germane  to  assessment 
and  utilization  of  results 

•  Identify  promising  approaches  for  assessment  and 
utilization  of  results 

•  Examine  validation  of  experiments 

•  Identify  the  critical  issues  in  assessing  experiments 
including: 

-  The  analysis  of  experiments  vs.  the  analysis  of  concepts 

-  Synthesis  of  experimental  results  for  DOTMLP 
recommendations 

•  Explore  our  ability  to  assess  experiments  (all  three 
levels  -  joint,  major  leveraged,  minor  leveraged) 

•  Suggest  actions  that  need  to  be  taken  to  improve  our 
ability  to  undertake  the  kind  of  assessment  program 
needed  for  joint  experimentation 


The  objectives  were  further  refined  by  choosing  a  series  of  specific  tasks  from  the 
overall  Terms  of  Reference  (TOR)  for  the  meeting  and  modifying  them  to  fit  the 
working  group  theme. 
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Assessment  and  Utilization  of  Results  Working  Group 

Agenda  of  WG  Presentations 


•  Tues,  9  March 

-  Introductions  -  Jacqueline  Henningsen,  FS 

-  ACOM  overview  -  CAPT  Oliver,  J97/J98 

•  Wed,  10  March 

-  Assessment  and  experimentation  -  LtGen  (ret)  Croker 

-  Service  experimentation 

•  JCFAWE  -  LTC  Kwan,  JCF  AWE  Assessments 

•  JEFX  -  Mr.  Schmidt,  AF  Experimentation  Office 

•  Thurs,  1 1  March 

-  Validation  of  results  -  Priscilla  Glasow 

-  Utilization  of  results  -  Mr.  Volmer 


Working  group  time  was  structured  by  the  use  of  a  limited  number  of  key  presentations 
focused  on  the  objective  areas.  This  included  a  challenge  briefing  from  CAPT  Oliver, 
J97/98.  Lt  Gen.  (ret)  Steve  Croaker,  USAF,  led  a  seminar  and  discussion  on  observations 
and  lessons  learned  from  EFX  98  with  broader  implications  for  assessments  of  experiments. 
Methods  for  including  assessment  in  the  Army  Advanced  Warfighting  Experiments  and  the 
Air  Force’s  next  Expeditionary  Air  Force  Experiment  were  presented  and  discussed  by  LTC 
Kwan  of  the  Army  and  Mr.  Schmidt  representing  the  Air  Force.  Priscilla  Glasow  led  a 
discussion  of  validation  as  it  applies  to  M&S  in  comparison  to  experimentation.  Finally, 

Mr.  Volmer  explored  the  utilization  of  experimental  results  by  external  groups  such  as 
Congress. 
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Assessment  and  Utilization  of  Results  Working  Group 

Key  Issues 

Identified  by  J97/J98,  Guest  Speakers  and  Participants 

•  Assessment  of  subjective/dirty  data 

•  2  levels  of  data  utilization  -  how  to  deal  with  each? 

-  Experiment  data 

-  Data  to  support  concepts 

•  Re-composition  of  information  to  ensure  complete 
coverage  of  a  concept 

•  Who/what/where/when/why/how  of  assessment  and 
who  is  the  customer 

•  Validation:  similarities  and  differences  of  meaning  for 
experimentation  and  M&S 

•  Best  utilization  of  experimental  results  for 
public/private  debate 


On  the  first  afternoon.  Captain  J.D.  Oliver,  J97/98,  presented  an  overview  of  the  J97 
Assessment  Division  and  the  J98  Integration  Division.  He  identified  the  following  issues  for 
working  group  discussion: 

1)  Assessment  of  subjective/dirty  data. 

2)  Experiment  versus  conceptual  data  utilization. 

3  Recomposition  of  information  to  ensure  complete  coverage  of  a  concept. 

To  these  baseline  issues,  the  group  added  the  requirement  to  understand  “who,  what,  where, 
when,  why  and  how  of  assessment  and  who  is  the  customer. 

Further  progress  through  the  presentations  led  to  concerns  about  the  utilization  of 
experimental  results  in  the  public  (external  to  the  experimental  process)  and  private  (relevant 
to  the  experimental  purpose.) 
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Assessment  and  Utilization  of  Results  Working  Group 

Assessment  Issues 


•  Ethical  issues  in  experimentation  -  what  are  laws  for 
experimentation  with  people,  do  they  apply  to  military? 

•  Managing  the  media  with  respect  to  results 

•  Misconception  that  assessment  is  an  ‘add  on’  to  the  experiment 

•  Participants 

-  Proper  Selection  Does  Matter  (Golden  Crews,  Cat-5  or 
Rainbow,  etc  ) 

-  Inclusion  (part  of  feedback  loop;  remediate  if  necessary) 

•  The  live  event  is  NOT  the  experiment;  it  may  not  even  be 
necessary 

•  Concept  exploration  should  include  many  experiments,  but 
experiment  may  be  the  universe,  part  of  an  exercise,  or  part  of 
a  broader  experiment 

-  How  does  assessment  differ  across  these?  The  tasks  are 
similar,  but  sphere  of  influence  differs 


Additional  issues  that  call  for  careful  consideration  are  provided  on  this  slide.  For  instance, 
members  were  surprised  by  the  information  that  regulations  may  call  for  careful  use  of  release 
processes  for  many  simple  experimental  assessment  procedures;  such  as  human  factors  based 
surveys.  The  group  noted  that  many  organizations  maintain  “human  subject  review  teams” 
and  suggested  this  may  need  exploration.  The  concept  of  effectively  managing  the  media  was 
not  intended  to  have  a  negative  connotation.  Rather  it  emphasized  the  idea  that  some  aspect  of 
assessment  preparation  may  need  to  be  focused  on  ensuring  that  a  reasonable  process  for 
providing  insights  and  first  order  observations  to  the  press  should  be  considered.  Participants 
stressed  the  idea  that  assessment  should  not  be  viewed  as  an  add-on,  but  treated  as  a  fully 
integrated  part  of  the  experiment.  The  group  spent  a  lot  of  time  discussing  the  criticality  of 
properly  selecting  the  participants  in  an  experiment.  It  is  extremely  important  to  know 
whether  the  use  of  uniquely  qualified,  not-fit  for  other  taskings  or  cross  conununity  teams  are 
best  suited  to  obtaining  experimental  information.  Equally  important  is  an  early  decision 
concerning  the  proposed  treatment  and  involvement  of  participants.  Will  they  be  full  partners 
in  the  assessment  process  with  transparency  in  results  or  will  they  be  shielded  from  knowledge 
of  outcomes.  If  a  major  problem  emerges  that  will  greatly  reduce  the  ability  of  obtaining 
valuable  data,  will  the  experiment  be  “restarted”  or  will  it  move  forward  at  all  costs.  Again, 
these  are  issues  that  need  to  be  decided  for  each  aspect  of  a  complex  experiment.  Working 
group  members  noted  that  it  is  very  important  to  dissuade  the  view  that  the  live  event  is  the 
valuable  or  only  real  part  of  the  experiment.  It  is  equally  important  to  use  the  full  pre  and  post 
period  activities  as  part  of  the  experiment  (possibly  by  the  use  of  simulation).  Focusing  on  the 
joint  experimentation  challenge,  participants  noted  that  concept  exploration  will  likely  be 
composed  of  numerous  experiments.  Therefore,  an  experiment  may  be  viewed  from  the 
perspective  of  how  it  fits  in  the  overall  campaign  or  how  it  is  meeting  a  narrow  objective.  The 
assessment  process  will  be  structured  differently  to  evaluate  the  context  of  the  experiment. 


83 


Assessment  and  Utilization  of  Results  Working  Group 

Validation  Issues 


•  Validation  of  simulation  was  the  discussion  point  of  departure 

•  WG  explored  how  validation  might  fit  experimentation. 

•  Issues 

-  Validation  planning  occurs  simultaneously  with  experiment 
concept  development  and  planning 

-  Quaiitative  and  quantitative  methods  will  be  used  in 
experimentation  and  validation  must  address  both 

-  Determination  of  organizational  roles,  responsibilities,  and 
relationships  is  a  prerequisite  to  experiment  validation 

-  Each  experiment  is  a  different  assessment  event  requiring 
different  validation  requirements  and  approaches 


Priscilla  Glasow’s  guided  discussion  on  validation  led  to  the  identification  of  the  issues  listed 
above.  It  was  viewed  as  important,  but  under-appreciated,  that  validation  planning  should 
occur  simultaneously  with  experiment  planning.  Further,  the  validation  must  address  both 
the  qualitative  and  quantitative  aspects  of  the  experiment.  The  former  is  obviously  easier 
than  the  latter.  It  is  equally  important  to  understand  that  validation  does  not  cut  across  all 
assessment  events  in  an  experiment.  It  must  be  tailored  to  each  aspect  appropriately. 
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Assessment  and  Utilization  of  Results  Working  Group 

Utilization  of  Results  Issues 

•  Results  published  on  front  page  of  the  Post  ( w/o  approval) 

•  Wrong  lessons  learned/political  filtering 

•  Perishable  data/results  (timeliness) 

•  Results  don’t  get  to  users  (right  levels) 

•  Loss  of  capture  opportunities 

•  Roles  and  missions  (Title  10  Issues) 

•  Failure  of  experiment  to  identify  all  relevant 
issues  (e.g.  logistics/information/training) 

•  Service  concerns  (re:  loss  of  destiny  control,  loss  of  control  of 
experiments) 

•  Ownership  of  data  (DoD  control  and  persistence  —  don ’t  pay 
for  data  twice,  know  where  to  get  data) 


The  utilization  of  results  issues  focused  on  the  public  vice  private  use  of  experimental 
results.  The  latter  is  most  frequently  what  the  designers,  planners  and  players  have  in 
mind,  but  the  former  should  be  given  careful  attention.  Demgers  lie  not  only  in  early  or 
inaccurate  release  of  results,  but  equally  with  the  possibility  of  negative  lessons  learned. 
Without  proper  training  in  the  purpose  of  an  experimental  design,  participants  frequently 
fall  into  the  “win  at  all  costs”  exercise  mentality.  This  can  not  only  lead  to  poor 
experimental  data  gathering,  but  also  to  negative  learning  and  disgruntled  participants  who 
report  their  limited  perception  as  truth  for  the  whole  experiment.  This  can  also  occur 
when  observers  see  a  weakness  or  “failure.”  The  ability  to  fail  is  the  mantra  of 
experimentation,  but  is  difficult  to  communicate  as  a  learning  tool.  A  number  of  other 
important  considerations  in  the  utilization  of  results  are  noted.  The  presentation  by  Chuck 
Volmer  highlighted  a  wide  range  of  additional  linkages  to  areas  as  broad  reaching  as  the 
National  Security  structure  debate. 
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Assessment  and  Utilization  of  Results  Working  Group 

Lessons  Learned  Tutorial 

_ by  Lt  Gen  (ret)  Steve  Croker 

•  Designing  the  Experiment 

-  Need  a  high  level  operations  concept,  end  state 

-  Prioritization  scheme  critical  up  front 

-  Must  enforce  a  “good  idea”  cutoff  date 

-  Experiment  Director  needs  strong  authority 

-  Need  Service/Joint  policy  for  experiments 

•  Planning  the  Assessment 

-  Decide  early  who  is  doing  the  assessment 

-  Integrate  assessors  as  full  partners 

-  Publish  one  set  of  collective  lessons  learned 

-  Remember  experiments  are  NOT  exercises 

-  Focus  on  assessor  manning,  stop  turnover 

-  Select  assessors  from  community  of  operators,  developers  and 
testers 


Lt  Gen  Steve  Croker,  US  AF  (ret),  who  has  served  as  a  Senior  Mentor  for  numerous  service 
and  joint  experimentation,  wargaming  and  exercise  events,  led  a  seminar  drawn  from 
observations  of  Expeditionary  Air  Force  Experiment  (EFX  98)  and  other  venues  to  explore 
lessons  learned,  key  assessment  issues,  data  collection  methods  and  an  assessment  roadmap. 
He  emphasized  that  large-scale  experimentation  is  a  new,  uncertain  process  and  it  is 
essential  to  allow  for  failures.  EFX  98  validated  the  promise  of  experimentation  and 
provided  a  test  bed  for  evolving  the  experimentation  assessment  process.  He  focused  on 
some  significant  assessment  lessons  included  the  value  of  using  an  ‘‘enterprise”  versus  an 
“independent  test”  approach  to  assessment.  (He  noted  that  even  more  fundamental  is  the 
importance  of  clearly  communicating  to  all  parties  the  underlying  premise  of  whatever 
assessment  process  will  be  used). 
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Assessment  and  Utilization  of  Results  Working  Group 

Lessons  Learned  Tutorial  (continued) 

by  Lt  Gen  (ret)  Steve  Croker 

•  Implementing  the  Assessment 

-  Keep  all  assessments  under  one  roof 
-10:1  operator  to  assessor  ratio  about  right 

-  Rate  the  experience  of  the  source 

-  Cross-functional  assessment  teams  work 

-  Need  a  simple,  manual  questionnaire 

-  Don’t  forget  the  key  people’s  Judgements 

•  Process  Notes 

-  When  assessing  tools  to  be  used  by  all  services,  use  all  services 
equipment 

-  The  ability  of  M&S  to  stop/start  and  provide  “instant  repiays”  is  key 

-  Test  piece  parts  before  the  entire  system 

-  Unless  we  know  where  we  are  going,  any  road  wiil  get  us  there 


The  enterprise  approach  required  that  assessors  are  considered  full  partners  in  the  experimental 
process,  and  are  integrated  into  the  experiment  from  the  first  step  of  planning.  It  is  also  important 
to  recognize  the  need  for  early  involvement  of  M&S,  the  establishment  of  prioritization  schemes 
to  help  distinguish  the  relative  importance  of  various  experimental  components,  and  to  realize  the 
difficulty  of  layering  experiments  onto  exercises.  A  number  of  his  discussion  topics  are  provided 
on  these  slides.  Additional  insights  were  included  in  the  full  presentation. 
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Assessment  and  Utilization  of  Results  Working  Group 

Assessment:  Who 


•  Team  with  “strong”  lead  analyst  —  “junk  yard  dog” 

-  Joint  representation  (coalition?) 

-  Multidisciplinary 

-  Subject-matter  expertise 

-  Analytic  expertise 

-  Data  technicians 

•  Evidence  of  success 

-  Credible  to  range  of  customers 

-  Team  perceived  as  knowledgeable 

-  Authority  is  sufficient  to  ensure  balance  between 
different  levels  of  experiment 


Various  aspects  of  “who”  should  lead  and  participate  in  the  assessment  team  was  explored. 
Participants  were  amused  by  one  speaker’s  statement  that  the  leader  must  have  the  tenacity 
of  a  “junk  yard  dog”  in  order  to  keep  the  balanced  position  for  assessment.  The  proper 
representatives  on  an  assessment  team  are  not  uniformly  identifiable,  but  the  notes  on  this 
page  indicate  some  of  the  considerations  that  should  be  part  of  the  planning  process.  The 
Assessment  planners  may  wish  to  develop  a  checklist  that  is  reviewed  for  each  “event.”  A 
balanced  team  may  need  to  have  a  multidisciplinary  rainbow  coalition  of  its  own  (similar 
to  the  EFX  98  concept  of  participants  selected  from  a  range  of  venues).  The  role  of  joint 
and  coalition  membership  on  the  assessment  team  vice  on  an  observer  team  will  be  closely 
linked  to  the  objectives  of  the  event.  Clearly  subject  matter  expertise  is  desirable,  but 
there  is  also  the  difficult  consideration  of  whether  to  put  the  best  SME’s  in  player  or 
assessor  roles.  This  tension  can  be  addressed  if  the  assessment  is  considered  a  partnership 
rather  than  a  scored  test.  The  next  slide  addresses  this  concept. 
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How  Balance  of  Power  is  Different  for 
Experiments 


Experiment 

Training  ^ 


Operator: 

(Trainees)' 


Experiment 

Director 


The  graphic  in  this  slide  highlights  the  concept  introduced  by  Lt  Gen  Croker  and  expanded 
by  the  working  group  of  carefully  distinguishing  the  assessor  from  the  various  roles  of  an 
Assessor  in  training,  exercise  and  experimentation  venues.  The  heavy  lines  show  the  areas 
that  tend  to  have  greater  authority  in  each  venue.  Further  in  considering  each  case,  it  is 
important  to  understand  the  relationship  between  team  members.  In  experimentation,  the 
group  hypothesized  that  assessor  (little  a)  should  be  an  equal  partner  in  a  team  that  includes 
the  operators,  and  the  developers  (concept  or  system).  An  Assessor  (big  A)  in  the 
traditional  experimental  sense  stands  back  (with  a  clipboard)  and  grades  the  experiment 
while  minimizing  interference.  In  the  case  of  the  assessor  (little  a),  assessment  takes  place 
through  a  cooperative  process  in  which  the  operators’,  and  when  appropriate  the 
developers’,  observations  are  captured  and  prepared  for  analysis  by  the  assessment  team 
using  various  data  capture  mechanisms.  In  this  case,  the  assessor  is  a  facilitator  of 
information  capture  and  appropriate  feedback.  The  group  postulated  that  there  is  no  trae 
experiment  value  without  integrated  assessment. 
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Assessment  and  Utilization  of  Results  Working  Group 


Balance  of  Power  for  Experiments 

Gen.  Croker 


Experimentation  Programs 


Continuing  the  graphic  introduced  in  the  previous  slide,  assessment  applies  not  only  to  an 
individual  experiment,  but  also  to  progress  on  the  overall  campaign.  In  view  of  the  use  of 
spiral  experimentation  to  advance  through  a  series  of  increasingly  complex  interactions,  the 
working  group  coined  the  term  spiral  assessment  to  capture  the  importance  of  tailoring  the 
assessment  process  to  different  stages  of  experimentation.  In  the  early  phases  of  exploration 
when  systems  are  just  being  introduced  or  concepts  being  refined,  the  assessment  process 
should  be  tightly  focused  on  identifying  ways  to  move  the  individual  aspect  along  (or 
identifying  its  inappropriateness  for  the  experiment.)  In  the  later  stages,  the  assessment 
should  refocus  on  the  interactions,  finally  in  the  final  stages  assessment  should  focus  on  the 
potential  of  the  system  of  systems  and  concepts.  The  term  layered  assessment  was 
introduced  to  focus  attention  on  the  need  to  provide  parallel  assessments  of  the  progress  of 
the  entire  Joint  Experimentation  campaign,  of  the  specific  aspects  of  various  experiments,  of 
the  individual  aspects  of  a  specific  experiment. 
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Leveraging  Other  Events 


Experiment 

Director 


Operato: 


Assessors 


Developers 


Joint  Experiment  Leveraging  Joint  Experiment  Leveraging 
a  Service  or  CINC  Experiment  an  Exercise 


The  pyramid  charts  were  used  to  portray  the  idea  that  current  programs  may  tend  to  be 
disjoint  experiments.  This  slide  highlights  the  point  that  there  are  multiple  layers  of 
relationship  between  different  events  and  the  overall  JE  Campaign.  For  instance,  a  joint 
experiment  may  be  embedded  within  a  service  or  CINC  exercise  or  experiment.  The  role  of 
the  assessors  in  this  case  must  clearly  indicate  a  reporting  chain  and  mles  of  procedure  for 
release  of  results. 
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Assessment:  What 


•  Assessment  integrated  with  all  phases  of 
experimentation: 

-  Concept  development:  evaluability  assessment;  meta¬ 
analysis  of  past  work 

-  Planning:  feasibility  assessment 

-  Design:  study  design,  scenario  development  and  data 
management  plan 

-  Conduct:  adaptable  data  collection  (prepare  for 
unexpected);  feedback  (modification;  rerun...) 

-  Assessment: 

•  Post-1:  immediate  assessment  reporting 

•  Post-2:  integrative  assessment:  synthesis;  knowledge 
management  (storage  and  retrieval  for  future  use)  —  may  require 
‘iterative  reporting’ 

-  Integration:  re-composition  of  information  to  assess 
concepts  across  multipie  events 


Assessment  must  be  fully  integrated  throughout  all  phases  of  experimentation  including 
concept  development,  planning,  design,  as  well  as  conduct.  Assessment  is  so  vital  to  the 
outcomes  and  the  resources  involved  in  experiments  are  so  dear,  that  assessors  should  be 
fully  integrated  into  the  experimental  design,  planning,  execution  and  replanning  and 
have  representation  at  the  senior  level  of  the  oversight  process.  The  working  group 
concluded  that  aloof  independence  is  not  preferred  at  the  level  of  the  actual  experiment 
and  that  operators  and  developers  are  part  of  the  assessment  team.  There  is  a  need  to 
establish  the  credibility  of  the  assessment  team  interface  and  a  strong  support  for  the 
assessors  needs  to  be  established  early  in  the  process.  This  means  that  Rules  of 
Engagement  (RoE)  and  interaction  will  need  to  be  carefully  defined  by  a  strong 
assessment  team  lead  who  is  tightly  linked  to  the  Experiment  Director  (ED). 
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Assessment:  What  (continued) 

•  Evidence  of  Success: 

-  Appropriate  scope:  asking  the  right  questions  in 
context 

-  Accurate  and  “objective”  with  quantitative  and 
qualitative  data  analysis 

-  Full  disclosure  in  reporting 

-  Results  oriented,  for  example 

•  Impact  on  force 

•  Candidates  for  acquisition 

•  DOTMLP  impact 

•  Experiment  and  analysis  implications 


The  group  examined  various  measures  of  what  should  be  included  in  assessment.  Members 
were  particularly  concerned  that  the  assessment  process  be  results  oriented.  Very  rapid 
feedback  concerning  progress  on  objectives  has  been  observed  as  a  standard  expectation  of 
senior  leadership,  and  participants  as  well  as  observers  will  freely  disseminate  their  views  of 
success  or  failure,  so  a  carefully  prepared  process  for  reporting  various  levels  of  the 
assessment  in  a  timely  manner  is  critical.  If  an  assessment  plan  with  timed  feedback  points 
and  level  of  detail  is  part  of  the  original  plan  as  well  as  steps  for  transitioning  the  results  into 
future  events  is  prepared  during  the  design  and  planning  phase,  there  is  more  likelihood  that 
valuable  lessons  will  not  be  overlooked.  A  key  point  noted  time  and  again  was  the  need  to 
instill  at  all  levels  that  an  EXPERIMENT  IS  NOT  THE  SAME  AS  AN  EXERCISE. 
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Assessment:  What  (concluded) 


•  Issues: 

-  Assessment  tasks  are  shared  responsibility  with 
operators  and  developers 

-  Reporting  is  continuous  and  corporate/enterprise 
effort  (with  assessment  lead) 

•  Initial  impressions  reporting  from  rehearsals 
and  training  is  valid 

•  Immediate  reporting  before  participants  leave  a 
live  event  ensures  ‘proper’  lessons  learned 
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Assessment:  Where 


•  Multi-dimensions  of  assessment  (next  slide) 

•  Measures: 

-  Completeness  of  assessment  by  event 

-  Completeness  of  assessment  by  concept 

•  Issues: 

-  Very  complex 

-  Re-composition  of  information  across  events  to 
support  concepts 


Measures  for  assessment  need  to  be  developed  and  responsive  to  not  only  the  various 
concepts  being  explored,  but  also  to  the  various  event  types  such  as  joint,  major  leveraged 
(usually  multi-Service  initiated  events)  and  minor  leveraged  (perhaps  a  tag  on  to  an 
individual  Service  event.)  Decisions  about  the  location,  distribution,  housing,  funding, 
reporting  chain  and  other  aspects  of  the  assessment  process  and  team  must  be  made. 
Experienced  group  members  cautioned  against  removing  the  assessors  too  far  from  the 
“heart”  of  the  experiment  and  the  ability  to  quickly  identify  and  work  with  the  director  to 
resolve  emerging  problems  that  would  minimize  the  ability  to  work  on  core  objectives. 
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Assessment:  When 


•  Early  and  continuous:  start  with  concept 

•  Immediate  utility 

•  Cumulative  utility 

•  Measures: 

-  Timely  information 

-  Knowledge  repository 

•  Issues: 

-  Analysts  must  find  ways  to  be  timely  without 
sacrificing  quality 


It  is  worth  repeating  that  assessment  must  be  fully  integrated  into  the  experimental  design, 
implementation  and  review.  Both  aspects  of  immediate  and  cumulative  utility  must  be 
considered  in  the  design  of  the  assessment  process.  Measures  must  account  for  both  the 
release  of  timely  information  sufficient  to  satisfy  early  needs  for  feedback,  and  the 
requirements  for  a  more  comprehensive  knowledge  repository  that  can  be  used  to  develop 
an  encompassing  picture  of  progress  toward  the  JE  campaign  goals.  It  is  critical  that  the 
assessors/analysts  find  ways  to  be  both  timely  and  to  maintain  quality.  The  partnership  with 
the  whole  exercise  team  must  be  attuned  to  fulfilling  this  critical  function. 


Assessment  and  Utilization  of  Results  Working  Group 

Assessment:  Why 


•  Experiment  is  equivalent  to  Assessment  (analysts  ’ 
relief  act) 

•  Experiment  is  the. . . 

-  Act  of  attempting  an  innovative  method  of 
operation,  especially  to  assess  its  feasibility, 
evaluate  its  utility  or  determine  its  limits. 

-  Opportunity  to  observe  military  phenomena  and 
gather,  interpret  and  act  on  information. 

•  Without  assessment,  an  experiment  is  just  a 
demonstration. 


It  is  easy  to  come  to  a  conclusion  that  experimentation  is  inseparable  from  assessment 
(group  members  joked  that  an  understanding  of  the  absolutely  critical  role  of  analysts  in 
this  process  would  lead  to  renewed  appreciation  of  the  role  of  the  analyst).  This  slide 
identifies  some  of  the  ways  that  an  experiment  (in  the  context  of  joint  experimentation) 
can  be  considered.  Some  of  the  ways  that  experimentation  success  can  be  measured  are 
the  longevity  of  the  findings,  the  discovery  of  new  ideas  and  the  amount  of  value  added 
information  for  decision  makers.  Some  issues  include  the  lack  of  common  definitions 
which  may  lead  to  false  expectations,  and  the  inability  to  separate  events  (such  as 
exercises)  in  which  failure  is  unacceptable  from  experimentation  where  failure  is  often 
the  source  of  the  most  important  lessons. 
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Assessment  and  Utilization  of  Results  Working  Group 

Assessment:  Why  (concluded) 


•  Measures: 

-  Longevity 

-  Discovery,  new  ideas 

-  Value  added  information  for  decision  making 

•  Issues: 

-  Lack  of  common  definitions  (understanding)  for 
experiment  and  assessment  (possibly  unrealistic 
expectations) 
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Assessment:  How 


•  Spiral:  iterative  process 

•  Methods  for  deconfliction  and  determining 
interaction/integration  among  concepts  required 

•  Tools  for  data  collection,  reduction,  analysis, 
synthesis  —  possible  requirement  for  new  tool 
development  to  support  experimentation 

-  Models  and  simulations 

-  Quantitative  and  quaiitative 

•  Measures  of  merit:  MOP/MOE,  EOS,  KRA 

•  Open  to  discovery;  expect  unexpected;  flexible  and 
adaptive 

•  Resources 


Several  examples  of  how  to  ran  experiments  were  examined.  Some  features  that  the  group 
felt  were  valuable  to  the  assessment  process  included  the  concept  of  SPIRAL  assessment 
that  was  previously  described  in  the  pyramid  charts.  Spiral  assessment  processes  may  help 
work  through  the  need  to  deconflict  and  examine  the  interactions  among  the  concepts  that 
are  under  review  both  globally  and  for  a  specific  experiment.  It  is  possible  that  new  tools 
for  data  collection  reduction,  analysis  or  synthesis  will  be  required  to  support  the 
experimentation  process,  but  funding  issues  are  more  likely  to  encourage  the  use  of 
existing  or  analyst  developed  adaptations  of  existing  tools.  Still,  the  assessment  process 
should  not  be  left  without  the  resources  to  develop  critical  tools,  to  bring  on  board 
appropriate  experts  or  to  maintain  continuity  of  involvement  with  experienced  team 
members. 
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Assessment  and  Utilization  of  Results  Working  Group 

Assessment:  How  (concluded) 


•  Measures: 

-  Useful  information 

•  Issues: 

-  Assessment  mis-perceived  as  add-on  to  an 
experiment  —  must  have  integration  from  ‘day  one’ 

-  Subjective/dirty  data  methods  (USAF  methods 
depicted  on  following  3  slides) 
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Assessment:  Customers 


•  Direct: 

-  JROC/CJCS 

-  OSD 

-  Services 

-  CINC  Commanders 

-  Participants 

•  Indirect: 

-  Congress 

-  Agencies 

-  Industry 


The  idea  that  there  are  both  direct  and  indirect  customers  for  experimentation  is  an  essential 
understanding  for  the  assessment  planning  process.  If  the  assessment  is  not  developed  in  a 
way  to  address  both  requirements,  then  contradictory  information  may  emerge.  The 
measures  for  the  “prime”  customer  may  differ  from  the  measures  needed  for  a  public  affairs 
presentation.  The  JE  leadership  must  determine  early  in  the  process  what  the  relationship 
will  be  between  the  various  members  of  the  team  in  terms  of  use,  release  and  interpretation 
of  information.  The  analyst  usually  prefers  to  keep  a  distance  from  this  aspect  of  use  of 
information,  but  it  is  valuable  to  note  that  an  information  void  will  be  filled  by  pieces  of 
information  expanded  to  fill  holes.  Once  a  perception  is  formed,  it  is  often  difficult  to  show 
its  weakness  with  later  arriving  information.  This  aspect  of  assessment  was  further 
explored  in  the  section  on  use  of  results. 


101 


Assessment  and  Utilization  of  Results  Working  Group 

Assessment:  Customers  (concluded) 


•  Measures: 

-  Satisfaction  by  prime  customer 

•  Does  not  equate  to  ‘liking  the  results’ 

-  Use  of  knowledge  gained 

•  Issues: 

-  Differing  (conflicting?)  expectations  (uses) 

-  Mis-use  of  results  by  customer  or  for  unintended 
customers 
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Validation  of  Experiments 


•  Concerns 

-  Need  to  maintain  focus  on  the  analyticai  purpose  of  the 
experiment  and  validate/assess  against  that  purpose. 

-  Modeis  and  simulations  embedded  for  use  within  an 
experiment  must  be  verified,  validated  and  accredited  for  that 
intended  use. 

•  Recommendations 

-  OR  organizations  that  support  experimentation  might  offer 
refresher  training  in  the  use  of  qualitative  methods  and  how 
to  validate  and  integrate  qualitative  data  with  quantitative 
results. 

-  Definition  of  validation  concepts  as  they  apply  to 
experimentation  is  needed  and  may  benefit  from  a  follow-on 
MORS  workshop. 


Priscilla  Glasow  of  the  Mitre  Corporation,  who  is  an  expert  on  DoD  Modeling  and 
Simulation  Verification,  Validation  and  Accreditation,  led  the  group  in  an  exploration  of  the 
similarities  and  differences  between  M&S  and  experiment  validation.  Participants  focused 
on  the  use  of  analytic  tools  and  models,  the  producing  of  “actionable”  information  and  the 
fact  that  both  offer  the  opportunity  to  empirically  observe  military  phenomena.  Differences 
include  the  fact  that  simulation  examines  potential  effects;  experimentation  examines  actual 
effects;  and  simulation  applications  are  carefully  constructed  processes  that  are  used  to  test 
hypotheses.  In  contrast,  experimentation  (as  defined  by  Worley)  is  a  heuristic  approach  that 
employs  trial  and  error  to  support  discovery.  Some  specific  concerns  and  recommendations 
are  shown  on  this  slide. 
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Utilization  of  Results 

•  Invest/divest  (revise  -  fine  tune  investment  strategy) 

•  Evolve  DOTMLP  recommendations 

•  Document  the  analytical  underpinning  of  results 

•  Marketing  (pro/con) 

•  Guide  industry  partners/R&D 


Chuck  Volmer,  VII  Inc,  presented  a  thought  provoking  discussion  on  the  utilization  of 
experimental  results  to  the  working  group.  He  started  with  the  hypothesis  that  “DoD’s 
ORSA/Experimentation  process  may  lack  management  and  administrative  checks  and 
balances  to  avoid  intentional  or  unintentional  manipulation  or  misuse  of  analytical  results.” 
He  noted  that  experiments  run  the  gamut  from  studies  to  live  fire  demos.  Mr.  Volmer 
discussed  the  distinctions  between  the  private  (internal  to  DoD,  a  service  or  a  sector)  and  the 
private  national  security  debates.  He  noted  that  joint  experimentation  as  a  part  of  national 
security  public  debate  has  been  offered  as  a  big  part  of  the  solution  to  arriving  at  the  answers 
as  well  as  a  catalyst  for  transformation  (by  the  National  Defense  Panel  (NDP)).  He 
concluded  that  DoD’s  path  to  2025  will  depend  largely  on  the  integrity  and  discipline  of  the 
ORSA  and  Experimentation  process.  A  sample  of  some  of  the  detailed  information  covered 
in  this  presentation  includes;  an  overview  of  the  over  1 25  “experiments”  a  year  conducted  by 
DoD,  a  matrix  for  managing  complex  change,  an  examination  of  some  of  the  challenges  to 
organizational  change  and  a  call  for  a  single  strategic  framework  to  integrate  studies, 
wargames/seminars,  exercises  and  experimentation.  His  suggestions  for  utilization  of 
experimental  results  are  found  on  slide  33,  but  it  is  recommended  that  the  full  presentation  be 
examined  by  those  who  are  interested  in  a  broader  understanding  of  this  important  aspect  of 
experimentation. 
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Utilization  of  Results  (concluded) 


•  Identify  discoveries  and  insights  for  further  exploration 

•  Next  event  planning 

•  Refine  experiment  methods 

•  Ensure  results  become  part  of  the  DoD  permanent 
body  of  knowledge 

•  Exploit  LHF 

(Low  Hanging  Fruit) 
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DoD  Conducts  Over  125  Experiments 
per  Year 


VII  Study  &  Database 


375  DoD  Events  (FY97-99):  _ 

Studies  end  Anelyses  synthetic  Environmt 

Wargames 

Modeling  and  Simulation 
Exercises  . 

Advanced  Concept  Technology 
Demonstrations 
Battle  Laboratories 
Organization  (Emphasis) 

153  Army  Events  (Modeling  and 
Simulation) 

120  Joint  Events  (Decision  Supporn^ 

“>*  71  Air  Force  Events  (Studies  and  iiij. 

Analyses)  L__ _ 

20  Marine  Corps  Events  (Live 
Demos) 

<^11  Navy  Events  (Wargames) 

DoD  Needs  To  Avoid  “Synthetic  Environment  Overload’ 
As  We  Collectively  Approach  Major  Policy  Milestones 


VII,  Inc 
Volmer 
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A  DoD  Oversight  Group  Might  Be  Beneficial  For  Assessment 
Validation.  Integration  And  Adjudication  Of  Joint  and  Inter- 
Department/Agency  Experiment  Results 
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Assessment  and  Utilization  of  Results  Working  Group 

Focus  On  Both  Public  And  Private 
Debates 


Private  Debate 

•  Battle-of-the-bullets 

•  Near-term  outlook 

•  Operations  driven 

•  Airpower  speak 

•  Service  focus 

•  System  orientation 

•  Emphasis 

-  “Shooters" 

-  “Sensors” 

-  “Movers” 


Current  DoD 

Public  Debate 

•  Battle-of-the-budget 

•  Far-term  outlook 

•  Policy  driven 

•  Joint/landA/Vashington 
speak 

•  Joint  focus 

•  System-of-systems 
orientation 

•  Emphasis 

-  “Sensors” 

-  “Movers” 

-  “Shooters” 


Results  Of  DoD  Experiments  Are  Often  Misused  In  The  Public  Debate 
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Suggestions 


Create  an  overall  strategic  framework  for  joint 
experimentation  and  inter-department/agency 
experimentation  (and  perhaps  intra-department 
experimentation) 

'"*■  Create  an  oversight  group  (other  than  ACOM)  to 
assess,  validate,  integrate  and  adjudicate  potential 
conflicts  before  they  are  aired  in  the  public  debate 
process 

Implement  quickly  (milestones  will  soon  be  upon  us) 
but  take  a  long-term  view 
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Additional  Insights 


•  AWE-LTCKwan 

•  EFX  -  Mr.  Schmidt 


In  the  words  of  Lt  Gen  Croker,  experimentation  (and  especially  large-scale 
experimentation)  are  essential  to  transform  the  DoD  into  a  relevant,  capable  21st  Century 
force.  Experimentation  IS  A  VALID  CONCEPT,  and  can  save  lots  of  money  and 
time.. .IF...  there  is  the  proper  assessment  and  validation  of  results  (so  it  is  not  just  a 
demonstration).  Experimentation  is  also  a  new  and  uncertain  method  where  all  the  best 
practices  have  not  yet  been  captured  and  catalogued.  Assessment  of  experimentation  is 
equally  new  and  uncertain,  where  all  the  best  practices  have  not  yet  been  captured  and 
catalogued.  So  a  certain  tolerance  for  failure  is  critical.  But  to  date,  the  promise  held  out 
where  experimentation  is  properly  supported  by  integrated  assessment  is  considerable. 
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AWE  Template  for  Analysis  Methodology 
(Model  -  Event  -  Model  [M-E-M]  Process) 


PRE-EXEROSE  ANALYSIS 


■  POST  EXERaSE  ANAt.YSTS 


^ —  exfor 


3HBSI; 

The  live  event 
is  NOT  the 
experiment; 
the  experiment 
is  the  entire 
process! 


1  \ 

— ? — 

— ““1 

, \ 

LTC  Michael  Kwan  described  the  Army  Advanced  Warfighting  Experiment  (AWE)  process. 
AWE’S  were  characterized  as  finding  ways  to  accelerate  change,  stimulate  thinking,  leverage 
judgements  of  commanders  and  soldiers  to  “change  the  way  [the  Army]  changes.”  AWE’s 
are  intended  to  reach  across  doctrine,  training,  leader  development,  organization,  materiel 
and  soldier  support.  The  focus  on  modekexercise-model  was  of  great  interest  to  participants. 
This  process  starts  with  scenario  and  issue  analysis  used  to  predict  the  outcomes  of  live 
simulations.  The  actual  outcomes  of  live  simulation  are  then  used  to  validate  and  expand 
“what  if’  analysis.  The  live  simulation  phase  is  used  to  focus  on  data  collection,  but  is  also 
frequently  enhanced  with  simulations/stimulations.  Four  recommended  parallel  steps  to 
prepare  for  a  successful  AWE  were  to  select  and  train  the  experimental  force,  the  opposing 
force,  the  observer/controllers  and  data  collectors  and  (educate)  DoD/service  senior  officials 
and  decision  makers.  In  particular,  he  provided  thoughts  on  the  Joint  Contingency  Force 
Advanced  Warfighting  Experiment  (JCFAWE).  He  explained  the  template  for  the  analysis 
methodology  that  outlines  the  model-exercise-model  process,  its  relation  to  the 
experimentation  process  and  a  strawman  analysis  approach.  He  stressed  the  importance  of 
the  pre-  and  post  exercise  period  in  which  simulation  and  modeling  is  used  to  evaluate  a 
broad  range  of  alternatives  that  are  impossible  to  exeimine  within  a  constrained  exercise 
venue. 
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EFX  ‘98  Assessment 


•  85  dedicated  assessors  and  300+  participants 

-  Located  at  distributed  rear,  forward  and  strategic  sites 

-  Focused  on  processes  and  technologies 

-  Run  through  entire  process  (3  Spirals  and  Experiment) 

-  Entered  directly  into  Joint  Universal  Lessons  Learned  (JULL)  database 

•  JFACC  “Hotwash^at  CORONA  Fall 

•  Technical  Assessment 


•  Web  Page  —  General  Observations  and  System  Application  Survey 


Mr.  Randy  Schmidt,  BETAC,  who  supports  the  Air  Force  Experimentation  Office 
(AFEO)  provided  additional  insights  on  the  Air  Force’s  EFX  process.  He  stressed  the 
importance  of  the  spiral  development  process  and  noted  that  the  experiment  mantra  should 
be  “input,  share  and  analyze.”  He  explained  an  evolving  web  technology  procedure  for 
gathering  experiment  lessons  learned  with  links  to  the  Joint  Universal  Lessons  Learned 
System.  He  noted  that  assessment  must  allow  extension  of  the  hypothesis  structure  from 
“if  -  then”  to  “if  -  then  -  so  what.” 
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JEFX’99  Data  Collection  Process 
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MILITARY  OPERATIONS  RESEARCH  SOCIETY 


Joint  Experimentation 
Workshop 

Report  of  the 
Synthesis  WG 


This  presentation  provides  the  highlights  from  the  deliberations  of  the  Synthesis 
Working  Group.  As  a  context  for  those  highlights,  a  business  process  re-engineering 
framework  is  introduced.  Using  that  framework,  the  report  focuses  on  six  key  areas: 
culture,  organization,  people,  processes,  resources  and  tools/data.  For  each  area,  key 
findings  and  recommendations  are  provided.  The  report  concludes  with  some  overall 
observations  on  joint  experimentation. 
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Working  Group  Participants 

•  Planning  Experiments 

Russ  Richards  (MITRE) 

•  Design  of  Experiments 

Dick  Hayes  (EBR) 

Clayton  Thomas,  PS  (USAF) 

•  Preparation,  Conduct  of... 

Ken  Jordan  (SAIC) 

•  M&S 

Dean  Free  (USN) 

Stuart  Starr  (MITRE) 

•  Assessment,  Utilization 

John  Baird  (Raytheon) 

LelandJoe  (RAND) 

Jerry  Kotchka  (Boeing) 

•  Floaters 

David  Alberts  (OSD) 

Col  Jon  Gallinetti  (ACOM) 

The  accompanying  vugraph  identifies  the  individuals  that  participated  on  the 
Synthesis  Working  Group,  It  can  be  seen  that  the  participants  brought  a  wide  variety 
of  perspectives  to  bear  on  the  problem  (i.e.,  joint  and  individual  Service;  industrial; 
FRRDC). 

Each  individual  on  the  working  group  had  two  assignments.  As  their  “Day  Job,”  they 
participated  on  the  individual  working  groups  cited  above.  As  their  “Night  Job,”  they 
had  two  tasks.  First,  they  helped  synthesize  selected  findings  and  recommendations  of 
the  individual  working  groups  into  a  “meta-story.”  Second,  they  provided 
independent,  value-added  findings  and  conclusions  to  this  synthesis  report. 


116 


To  provide  a  context  for  their  discussions,  the  Synthesis  Working  Group  adopted  the 
above  model  of  the  experimental  space.  The  space  is  defined  by  three  dimensions: 


•  The  maturity  of  knowledge  domain.  At  the  outset  of  an  experimental  campaign,  it 
is  presumed  that  relatively  little  is  known  about  the  new  operational  concept  of 
interest  (e.g.,  great  uncertainty  about  the  potential  effectiveness  of  the  concept 
over  a  broad  range  of  potential  scenarios).  By  the  conclusion  of  the  experimental 
campaign,  it  is  anticipated  that  senior  decision  makers  will  have  extensive 
knowledge  about  the  effectiveness  of  the  proposed  concept  and  an  in-depth 
appreciation  of  its  impact  on  Doctrine,  Organizations,  Training,  Materiel, 
Leadership  and  Personnel  (DOTML-P). 

•  Complexity.  There  are  many  dimensions  that  characterize  the  complexity  of  the 
candidate  concepts  of  operations.  These  include,  inter  alia,  the  numbers  of 
echelons  involved,  the  information  flow,  the  numbers  of  actors  and  their 
interactions.  As  suggested  by  the  figure,  it  would  be  prudent  to  begin  the 
experimental  campaign  by  conducting  experimental  events  that  are  limited  to 
simplified  aspects  of  the  concept  before  addressing  the  full  set  of  complicating 
factors. 

•  Tools.  Experimenters  have  a  broad  range  of  potential  tools  at  their  disposal 
ranging  from  low  fidelity  (e.g.,  expert  elicitation,  system  dynamics  models)  to 
high  fidelity  (e.g.,  live  experiments  in  complex  operational  environments).  As 
suggested  by  the  figure,  the  lower  fidelity  tools  should  be  used  to  develop  a  broad, 
initial  understanding  of  the  proposed  concept,  with  tools  of  greater  fidelity 
employed  selectively  to  gain  greater  maturity  of  knowledge. 
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A  Framework  for  Joint 
Experimentation _ 


Culture 


The  accompanying  figure  provides  a  business  process  re-engineering  perspective  of 
the  joint  experimentation  enterprise.  The  Synthesis  Working  Group  concluded  that  if 
joint  experimentation  is  to  be  successful,  it  must  address  all  of  these  factors, 
consistently. 

As  a  back  drop  to  the  effort,  consideration  must  be  given  to  the  cultural  changes  that 
are  a  necessary  prerequisite  to  effective  transformation  of  the  military  enterprise. 
Second,  there  are  several  organizational  issues  that  must  be  addressed.  These  involve 
both  the  organization  of  the  teams  charged  with  performing  experiments  and  the 
ACOM  J9  staff.  Third,  the  people  involved  in  all  phases  of  the  experiment  constitute  a 
critical  intellectual  resource.  Attention  must  be  paid  to  their  education  and  training  to 
prepare  them  to  participate  in  the  experimentation  process.  The  inner  segment  of  the 
framework  provides  the  linkage  between  high  level  guidance  to  the  experimental 
process  (e.g.,  policy  and  concepts)  and  the  primary  products  of  that  process  (i.e., 
DOTML-P  recommendations).  To  implement  this  linkage,  a  “spiral’'  set  of  actions  is 
envisaged  which  sequentially  performs  a  set  of  experimental  processes  supported  by  a 
broad  set  of  tools.  The  figure  emphasizes  that  these  experimental  processes  and  tools 
are  constrained  by  available  resources  (e.g.,  funds,  manpower,  systems).  The 
remainder  of  this  report  summarizes  major  findings  and  recommendations  for  selected 
elements  of  this  framework. 
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Culture 


Synthesis  WG 


•  Selected  Findings 

•  Joint  experimentation  is  only  one  of  many  elements  needed 
for  successful  transformation 

•  Major  obstacles  to  joint  experimentation  include 

•  Lack  of  common  lexicon 

•  Heterogeneous  views  about  nature  of  “Experimentation” 

•  Multiple  Time  Frames  (near-,  mid-,  far-term) 

•  Non-DoD  (e.g.,  coalition,  interagency,  NGO)  issues 
exacerbate  the  problem 


Selected  Findings.  The  Synthesis  Working  Group  found  that  joint  experimentation  is 
not  an  end  unto  itself.  It  is  only  one  of  many  elements  that  is  needed  to  transform  the 
military  enterprise  successfully.  As  an  illustration  of  some  of  the  other  critical  steps 
that  must  be  undertaken  in  concert  with  joint  experimentation.  Reference  1  cites 
nineteen  activities  (one  of  which  is  “analysis,  test,  simulation,  feedback  (early; 
repetitive)  and  lessons  learned”)  aggregated  into  the  categories  of  Concepts  and 
Analyses;  Organization  and  People;  Management  and  Direction;  and  Resources  and 
Technology. 

There  are  several  major  cultural  obstacles  that  must  be  overcome  if  the  joint 
experimentation  segment  of  the  transformation  is  to  be  executed  effectively  and 
efficiently.  First,  it  was  made  apparent  during  the  workshop  that  the  service  and  joint 
experimentation  communities  frequently  use  key  terms  (e.g.,  “experimentation”) 
inconsistently.  Second,  these  communities  have  heterogeneous  views  about  the  nature 
of  experimentation  (e.g.,  whether  they  are  on  “voyages  of  discovery,”  confirming 
proposed  hypotheses,  a  phase  of  the  “system-of-systems”  acquisition  process  or 
demonstrating  a  result).  Finally,  it  was  observed  that  confusion  is  introduced  by  the 
different  time  frames  of  interest  (e.g.,  near-term  explorations  of  variations  on 
operational  concepts,  vice  far-term  efforts  to  assess  basic  transformations  of  the 
military  establishment). 
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Culture  (Concluded) 

•  Selected  findings 

•  Joint  experimentation  is  only  one  of  many  elements  needed 
for  successful  transformation 

•  Major  obstacles  to  joint  experimentation  include 

•  Lack  of  common  lexicon 

•  Heterogeneous  views  about  nature  of  “experimentation" 

•  Multiple  time  frames  (near-,  mid-,  far-term) 

•  Non-DoD  (e.g.,  coalition,  interagency,  NGO)  issues 
exacerbate  the  problem 

•  Conclusions 

•  A  common  lexicon  must  be  developed  for 

•  Basic  terms 

•  Key  mission  areas 

•  Cultural  issues  must  be  addressed  at  the  leadership  level 


It  was  further  observed  that  future  experimentation  efforts  will  pose  still  more 
profound  cultural  changes.  As  experimentation  begins  to  turn  towards  the  issues 
associated  with  “New  World  Disorder”  missions  (e.g.,  peacemaking),  a  host  of  new 
participants  will  be  involved.  These  will  include  coalition  partners,  interagency 
participants  (e.g.,  the  Department  of  Justice  in  the  problem  of  critical  infrastructure 
protection)  and  nongovernmental  organizations  (e.g..  Red  Cross,  in  support  of 
humanitarian  assistance).  These  additional  perspectives  will  introduce  challenges  that 
will  make  it  more  difficult  to  execute  all  phases  of  an  experimental  campaign. 

Conclusions.  As  an  initial  step  towards  ameliorating  these  cultural  issues,  a  common 
lexicon  must  be  developed  for  the  basic  terms  used  to  describe  experiments  and 
experimental  campaigns.  In  addition,  this  lexicon  should  define  the  key  mission  areas 
that  are  the  subject  of  joint  experimentation  efforts.  MORS  is  well  positioned  to  take  a 
major  role  in  these  endeavors. 

Ultimately,  it  is  the  leadership  level  that  must  take  the  lead  in  effecting  the  many 
cultural  changes  that  are  required  if  successful  experimentation  is  to  result  in 
successful  transformation  of  the  military  enterprise. 
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Synthesis  WG 


•  Selected  findings 

•  Organizational  issues  have  degraded  the  utility  of  selected 
service  experiments;  e.g., 

•  Stovepiping,  lack  of  continuity,  lack  of  enterprise  perspective 

•  ACOM  J9  organization  appears  suitable  for  ramp  up;  perhaps 
sub-optimal  for  steady  state  operations 


Selected  findings.  There  are  two  important  facets  of  the  organizational  dimension  to  the 
experimentation  problem.  First,  it  was  noted  that  organizational  issues  have  degraded  the 
utility  of  selected  service  experiments.  For  example,  in  several  service  experiments  there 
has  been  a  lack  of  continuity  of  key  personnel  as  the  experimental  campaign  progressed. 
Those  personnel  discontinuities  have  adversely  affected  the  experimental  learning  curve. 

In  addition,  service  “stovepipes”  have  limited  the  transference  of  “lessons  recorded”  in  one 
service  experimentation  activity  to  another.  As  an  example,  during  the  Army’s  Task  Force 
XXI  experiments,  certain  lessons  were  recorded  that  were  briefed  to  the  USAF’s 
Expeditionary  Forces  Experiment  (EFX)  (e.g.,  limit  the  number  of  new  initiatives;  allow 
adequate  time  for  training  experimental  personnel;  enforce  a  “good  idea  cut-off  date”). 
However,  there  is  little  evidence  that  these  lessons  recorded  were  truly  learned  and  acted 
upon. 

The  second  facet  of  the  organizational  dimension  deals  with  the  organization  of  the 
ACOM  J9  staff.  Currently,  that  staff  has  adopted  an  organization  that  reflects  the 
individual  phases  of  the  experimentation  process  (i.e.,  planning,  design,  preparation, 
conduct,  assessments,.. .).  The  Synthesis  Working  Group  observed  that  this  organization  of 
the  staff  was  probably  suitable  for  the  initial  ramp  up  phase  of  J9.  However,  it  was 
observed  that  as  the  process  transitions  into  “steady  state”  (i.e.,  ACOM  pursues  several 
experimented  initiatives,  in  parallel),  it  may  prove  desirable  to  transition  to  a  matrix 
organization  where  life  cycle  responsibility  for  selected  experimental  themes  are  vested  in 
a  single  entity  with  support  from  cells  of  specialized  expertise. 
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Organization  (Conciuded) 

•  Selected  findings 

•  Organizational  issues  have  degraded  the  utility  of  selected 
service  experiments;  e.g., 

•  Stovepiping,  lack  of  continuity,  lack  of  enterprise  perspective 

•  ACOM  J9  organization  appears  suitable  for  ramp  up;  perhaps 
sub-optimal  for  steady  state  operations 

•  Conclusions 

•  A  focal  point  (e.g.,  ACOM  J9)  is  needed  to 

•  Collect  organizational  lessons  recorded”  in  service  experiments 

•  Distill  them  into  lessons  learned 

•  Facilitate  their  application 

•  Re-evaluation  of  ACOM  J9  organizational  structure  is  needed  to 
ensure  “cradle-to-grave”  continuity  (e.g,,  consider  matrix 
organization) 


Conclusions.  To  address  the  issues  that  emerge  from  organizational  stovepipes,  a 
focal  point  is  needed  to  collect  organizational  “lessons  recorded”  in  service 
experiments,  distill  them  into  “lessons  learned,”  and  facilitate  their  application  to  other 
joint  or  service  experiments.  ACOM  J9  is  uniquely  positioned  to  play  this  role. 

Finally,  as  noted  above,  the  benefits  and  costs  of  a  modified  ACOM  J9  should  be 
evaluated  in  which  “cradle~to~grave”  experimental  continuity  is  vested  in  an 
organizational  unit  with  matrixed  support  for  selected,  in-depth  technical  support. 
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People 


•  Selected  finding 

•  Education  and  training  are  deficient  for  all  participants 
•  Experiment  director,  the  consumers,  players,  assessors,  eta! 


Selected  Findings.  Several  presentations  at  the  workshop  revealed  that,  in  initial 
service  experiments,  education  and  training  have  been  deficient  for  all  participants  in 
experiments.  It  was  emphasized  that  this  deficiency  has  adversely  affected 
experimentation  performance  at  all  levels,  (i.e.,  from  the  experimental  director  through 
the  data  collectors/assessors).  One  of  the  reasons  for  this  shortfall  is  the  inevitable 
“time  cmnch”  that  is  the  result  of  “success-oriented”  planning  of  experiments.  Time 
for  education  and  training  is  generally  the  first  casualty.  As  a  consequence,  it  is  not 
unusued  for  the  players  to  be  inadequately  conversant  with  concepts  and  processes 
when  the  experiment  begins.  The  resulting  “on-the-experiment”  learning,  inevitably 
confounds  the  results  of  the  experiment. 


123 


i 

I 


Synthesis  WG 

People  (Concluded) 

•  Selected  finding 

•  Education  and  training  are  deficient  for  all  participants 

•  Experiment  director,  the  consumers,  players,  assessors,  et  al 

•  Conclusions 

•  An  education  campaign  is  needed  (with  courses  and 
materials) 

•  Sufficient  resources  and  adequate  time  are  needed  for 
training  to  performance  standards  (TBD)  prior  to  an 
experimental  event 

•  It  is  necessary  to  attract,  retain,  cross-train  and  enhance 
critical  skills 


Conclusion.  To  deal  with  this  critical  issue,  an  education  campaign  is  needed  to 
prepare  all  participants  in  the  experimental  process.  MORS  could  play  a  significant 
role  in  this  process  by  developing  suitable  courses  and  materials  and  delivering  them 
in  conjunction  with  the  annual  symposium. 

In  the  future,  sufficient  resources  and  adequate  time  for  training  must  be  planned  (and 
protected!)  prior  to  an  experimental  event.  It  is  recommended  that  the  criterion  for 
training  be  satisfaction  of  specified  performance  standards  vice  time  allocated  to 
training  (i.e.,  use  of  an  output  measure  versus  an  input  measure). 

Finally,  it  is  recommended  that  exceptional  steps  be  taken  to  attract,  retain,  cross-train 
and  enhance  the  critical  skills  needed  to  execute  experiments  and  experimental 
campaigns.  This  would  entail  a  number  of  personnel  actions.  First,  steps  should  be 
taken  to  make  an  experimental  assignment  an  attractive  one.  As  an  example,  it  would 
send  a  strong  message  if  key  performers  are  given  preferential 

promotions/assignments.  In  addition,  these  assignments  to  the  experimentation  process 
should  be  of  sufficient  duration  to  allow  key  individuals  to  spend  adequate  periods  of 
time  during  the  experimental  life  cycle. 
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Synthesis  WG 


•  Selected  findings 

•  Participation  has  not  always  been  either 

•  Continuous 

•  Universal 

•  Piggybacking  is  seductively  attractive,  but  is  very  difficult  to 
do  well 

•  Conclusions 

•  M  participants  need  to  get  involved  throughout  an 
experimental  campaign 


Selected  Findings.  Several  presentations  at  the  mini-symposium/workshop  noted  that 
participation  in  experiments  has  frequently  been  neither  continuous  nor  universal. 
Thus,  selected  participants  have  been  involved  in  the  experiment  sporadically,  to  the 
detriment  of  the  total  process.  For  example,  it  has  not  been  unusual  for  assessors  to 
become  involved  late  in  the  process,  adversely  affecting  their  ability  to  shape  and 
implement  assessment  activities. 

The  key  finding  in  this  category  deals  with  the  issue  of  “piggybacking.”  In  this 
concept,  a  joint  experiment  attempts  to  take  advantage  of  the  resources  that  have  been 
assembled  by  a  service  experiment,  to  satisfy  joint  objectives.  It  was  noted  that,  while 
this  concept  is  seductively  attractive,  it  is  very  difficult  to  do  well. 

Conclusions.  In  order  for  an  experiment  to  be  conducted  effectively  and  efficiently,  it 
requires  a  high  performance  team  to  work  together.  This  suggests  that  the  key 
participants  need  to  get  involved  throughout  an  experimental  campaign. 
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Piggybacking:  Concept  vs  Reality 

Concept 

Reality 

Joint 

The  above  cartoon  highlights  the  difference  between  the  piggybacking  concept  and  the 
reality.  In  the  concept,  the  joint  experiment  takes  advantage  of  the  resources 
assembled  by  a  Service  experiment  (without  adversely  affecting  the  Service’s  goals 
and  objectives). 

The  reality  tends  to  be  quite  different.  For  example,  Information  Superiority 
Experiment  (ISX)  1.1  sought  to  piggyback  on  EFX  98.  By  providing  $3.4M,  ISX  1.1 
more  than  doubled  the  live  fly  component  of  EFX  98,  thereby  enhancing  the  value  of 
the  latter  activity.  However,  limitations  imposed  by  EFX  98,  adversely  affected  the 
success  of  ISX  1.1  (see  reference  2). 

Reference  2  concluded  that  piggybacking  can  be  successful  only  if  both  parties  regard 
it  as  a  “win-win”  experience.  To  that  end,  they  derived  the  following  necessary 
conditions  for  piggybacking  success: 

•  High  level  visibility  and  leverage. 

•  Formal  agreement  to  collaborate. 

•  Agreement  that  piggybacking  is  equitable  for  all  participants  (“win-win”). 

•  Sufficient  influence  on  the  scenario  to  tailor  it. 

•  Early  involvement  in  the  planning  process  (e.g.,  deconfliction,  resource 
allocation). 

•  Sufficient  resources  for  training,  data  collection  and  analysis. 

•  A  robust  experimentation  environment;  including  adequate  number, 
variety  of  experimental  events,  instrumentation  and  free  play 
(opportunities  to  fail). 
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•  Selected  finding 

•  Resources  need  to  be  allocated  in  a 
balanced  fashion;  e.g., 

•  Pre-,  trans-,  post-experiment 

•  Training,  operations,  assessment,... 

•  Conclusion 

•  Key  activities  which  are  traditionally  under¬ 
funded  must  be  properly  resourced;  e.g., 

•  Pre-  and  post-experiment 

•  Training  and  assessment 


Selected  Findings.  In  order  to  perform  an  experiment  successfully,  it  requires  the 
balanced  performance  of  a  number  of  functions.  Temporally,  it  can  be  viewed  as 
balance  across  the  phases  of  an  experiment  (pre-,  trans-  and  post-experiment)  and  the 
individual  functions  that  must  be  performed  during  the  experimental  life  cycle  (e.g., 
training,  operations,  assessments...) 

Conclusions.  In  recent  experiments,  we  have  witnessed  the  failure  to  allocate 
resources  across  key  phases  and  functions  in  a  balanced  way.  For  example,  it  has  been 
reported  in  reference  2  that  the  pre-  and  post-experiment  phases  of  ISX  1 . 1  were 
underfunded  and  that  in  numerous  other  experiments,  training  and  assessment  were 
underfunded. 
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Tools,  Data 


Synthesis  WG 


•  Selected  findings 

•  No  single  tool  (or  class  of  tools)  is  adequate 

•  Key  voids  exist  in  our  tools,  data  and  knowledge  associated 
with  New  World  Disorder  issues 

•  Conclusions 

•  The  FAME  (Full  spectrum  Analysis,  Modeling  and  Simulation, 
and  Experiment)  paradigm  should  be  used 

•  A  mix  of  multi-resolution  tools,  with  associated  data,  is 
needed  to  support  joint  experimentation;  e.g., 

•  A  hierarchy  of  selected  community-accepted  M&S  (including 
new  tools  that  are  relevant  to  the  New  World  Disorder) 

•  “Boutique”  tools  that  can  be  created  rapidly  and  run  quickly  to 
help  focus  activities 


Selected  Findings.  There  is  a  tendency  to  equate  “experiment”  with  “live  Modeling  and 
Simulation  (M&S).”  However,  the  full  experimental  life  cycle  calls  for  multiple 
orchestrated  tools,  subsuming  expert  elicitation,  analysis,  system  dynamics  models, 
constructive  M&S,  virtual  M&S,  as  well  as  live  M&S.  In  addition,  as  we  begin  to  consider 
the  challenges  of  experimenting  with  new  world  disorder  issues  (e.g.,  critical  infrastructure 
protection,  peacemaking/peacekeeping),  it  becomes  apparent  that  key  voids  exist  in  our 
tools,  data  and  knowledge. 

Conclusions.  Consistent  with  the  recommendations  of  the  other  working  groups,  it  is 
agreed  that  a  full  spectrum  analysis,  Modeling  and  Simulation  and  Experiment  (FAME) 
paradigm  should  be  used. 

To  implement  this  paradigm,  a  mix  of  multi-resolution  tools,  with  associated  data,  are 
needed  to  support  joint  experimentation.  This  tool  set  should  contain  both  a  hierarchy  of 
selected  community-accepted  M&S  (including  new  tools  that  are  relevant  to  the  new 
world  disorder  and  a  set  of  “boutique”  tools  that  can  be  created  rapidly  and  run  quickly  to 
help  focus  activities. 
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Summary  (1  of  2) 


Synthesis  WG 


•  Clearly,  Joint  experimentation  is  in  its 
infancy... 


Based  on  the  presentations  that  were  given  at  the  mini-symposium  and  the  deliberations 
during  the  workshop,  it  is  clear  that  joint  experimentation  is  in  its  infancy. 
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One  manifestation  of  that  immaturity  is  suggested  by  the  above  cartoon.  At  this  point,  we 
have  many  theories  about  new  operational  concepts  that  could  conceivably  revolutionize 
warfare,  but  no  conclusive  proof  that  these  operational  concepts  will  realize  their  potential. 
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Summary  (1  of  2) 


Synthesis  WG 


•  Clearly,  Joint  experimentation  is  in  Its 
infancy... 

•  We  believe  that  joint  experimentation 
has  the  potential  to  illuminate  many  of 
the  contentious  issues  that  are  at  the 
heart  of  transforming  DoD... 


The  Synthesis  Working  Group  believes  that  joint  experimentation  has  the  potential  to 
illuminate  many  of  the  contentious  issues  that  are  at  the  heart  of  transforming  DoD. 
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For  example,  one  of  the  age  old  issues  in  DoD  is  the  proper  balance  between  the  quality  and 
quantity  of  weapon  systems.  The  above  cartoon  suggests  that  a  well-designed  experiment 
might  shed  some  light  on  that  issue. 
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Summary  (2  of  2) 


Synthesis  WG 


•  However,  successfully  implementing  all 
necessary  aspects  of  the  joint 
experiment  framework  will  NOT  be 

•  Easy;  e.g.,  successfully  dealing  with 

•  Planning  experimental  campaigns 

•  Educating  and  training  all  participants 

•  Piggybacking  equitably  and  effectively 

•  Rapid... 


However,  the  Synthesis  Working  Group  concluded  that  successfully  implementing  all 
necessary  aspects  of  the  joint  experiment  framework  will  NOT  be  easy.  Foremost 
among  the  obstacles  are  the  challenges  associated  with  successfully  planning 
experimental  campaigns,  educating  and  training  all  the  participants  in  an  experiment 
and  piggybacking  joint  experiments  equitably  and  effectively  on  service  experiments. 
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Hard  to  Prove  Concepts 


“It  may  very  well  be  the  key  to  immortality; 
but,  of  course,  it  will  take  forever  to  prove  it!” 


In  addition,  the  Synthesis  Working  Group  concluded  that  successfully  implementing 
all  necessary  aspects  of  the  joint  experiment  framework  will  NOT  be  rapid.  As 
suggested  in  the  above  cartoon,  the  complexity  and  dimensionality  of  the  issues  will 
require  a  long  term  commitment  before  the  decision  maker  will  have  sufficient 
information  so  that  he  can  transform  the  DoD  with  confidence. 

References: 

1.  “Management  Actions  and  Tactics  of  Innovation,”  Under  Secretary  of  Defense 
(Policy),  1995  Summer  Study,  Organized  by  the  Director,  Net  Assessment,  Newport, 
Rhode  Island,  31  July  -  9  August  1995. 

2.  “Lessons  Learned  from  Information  Superiority  Experiment  (ISX)  1.1,”  D.  Alberts, 
F.  R.  Richards,  &  S.  Starr,  Proceedings  of  1999  Command  and  Control  Research  & 
Technology  Symposium,  Newport,  RI,  29-30  June,  1  July  1999. 
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ACRONYMS 


ACOM 

ACTD 

AFEO 

AFTT 

AWE 

BOGSATs 

C^ISR 

CINC 

CJCS 

COBP 

CONORS 


DCINC 

DMSO 

DoD 

DoE 

DOTMLP 

DPG 

DUSD 

ED 

EFX 

EXFOR 

FAME 

FEDEP 

FFRDC 

FINPLAN 

HLA 

lAW 

EM 

ISX 

JCFAWE 

JE 

JROC 

JULL 

LAN 

LCM 

M&S 


Atlantic  Command 

Advanced  Concept  Technology  Demonstration 
Air  Force  Experiment  Office 
Airforce  Institute  of  Technology 
Advanced  Warfighting  Experiment 
Bunch  of  Guys/Gals  Sitting  Around  A  Table 
Command  and  Control 

Command,  Control,  Communication,  Computers,  Intelligence, 

Surveillance  and  Reconnaissance 

Commander-in-Chief 

Chairman  of  the  Joint  Chiefs  of  Staff 

Code  of  Best  Practices 

Concept  of  Operations.  A  verbal  or  graphic  statement,  in  broad  outline,  of  a 
commander's  assumptions  or  intent  in  regard  to  an  operation  or  series  of 
operations.  The  concept  of  operations  frequently  is  embodied  in  campaign  plans 
and  operation  plans;  in  the  later  case,  particularly  when  the  plans  cover  a  series 
of  connected  operations  to  be  carried  out  simultaneously  or  in  succession.  The 
concept  is  designed  to  give  an  overall  picture  of  the  operation.  It  is  included 
primarily  for  additional  clarity  of  purpose. 

Deputy  Commander-in-Chief  (BMDO) 

Defense  Modeling  and  Simulation  Office 
Department  of  Defense 
Design  of  Experiments 

Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Personnel 

Defense  Planning  Guidance 

Deputy  Under  Secretary  of  Defense  (BMDO) 

Experiment  Directive/Experiment  Director 
Expeditionary  Forces  Experiment 
Exercise  Force 

A  Full  spectrum  Analysis  employing  Modeling  techniques,  tools  and 
Experiments 

Federal  Execution  and  Development  Process 

Federally  Funded  Research  and  Development  Plan 

Financial  Plan 

High  Level  Architecture 

In  Accordance  With 

Initial  Insights  Memorandum 

Information  Superiority  Experiment 

Joint  Contingency  Force  Advanced  Warfighting  Experiment 

Joint  Experimentation 

Joint  Requirements  Oversight  Council 

Joint  Universal  Lessons  Learned 

Local  Area  Network 

Life  Cycle  Management 

Modeling  emd  Simulation 
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M-E-M 

Model-Exercise-Model 

MLS 

Multiple  Levels  of  Security 

MOE 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 

MORS 

Military  Operations  Research  Society 

MOUs 

Memorandum  of  Understanding 

MOUT 

Military  Operations  in  Urban  Terrain 

M-T-M 

Model-Test-Model 

NDP 

National  Defense  Panel 

NPS 

Naval  Postgraduate  School 

OASD 

Office  of  Assistant  Secretary  of  Defense 

OOTW 

Operations  Other  Than  War 

OPFOR 

Opposing  Force 

ORSA 

Operations  Research/Systems  Analysis 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

Operational  Test  and  Evaluation 

OTA 

Operational  Test  Agencies 

R&D 

Research  and  Development 

ROI 

Return  on  Investment 

SME 

Subject  Matter  Experts 

T&E 

Test  and  Evaluation 

TOR 

Terms  of  Reference 

UFR 

UnFunded  Resources 

USACOM 

US  Atlantic  Command 

VIP 

Very  Important  Person 

WA 

Verification,  Validation  and  Accreditation 

RoE 

Rules  of  Engagement 

WMD 

Weapons  of  Mass  Destruction 
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Military  Operations  Research  Society 
Joint  Experimentation  Mini-Symposium  and  Workshop 
Terms  of  Reference 
2  February  1999 


Background 

The  Revolution  in  Military  Affairs  is  transforming  not  only  our  technological  capabilities,  but 
also  our  organizational  structure,  doctrine,  leadership  and  training  requirements.  The  pace  of 
change  is  too  fast  for  us  to  wait  until  the  technologies  are  in  place  before  learning  how  to  exploit 
them.  Experiments  are  a  way  to  bridge  this  gap.  Well  designed  and  conducted  experiments  can 
point  the  way  to  the  required  organizational,  doctrinal  and  cultural  changes  that  best  take 
advantage  of  the  opportunities  offered  by  advancing  technologies. 

Experimentation  will  play  a  pivotal  role  in  our  journey  to  Joint  Vision  2010  and  beyond. 
Effective  1  October  1998,  the  Commander-in-Chief,  U.S.  Atlantic  Command  (CE^CUSACOM) 
is  the  Defense  Department’s  Executive  Agent  for  Joint  Experimentation.  CINCUSACOM  has 
the  authority  and  responsibility  for  developing  and  implementing  an  aggressive  program  of 
experimentation  to  foster  innovation  and  rapid  fielding  of  new  concepts  and  capability  for  joint 
operations.  The  Joint  Experimentation  Implementation  Plan  (IPLAN)  establishes  the 
experimentation  process  and  explains  how  the  USACOM  Joint  Experimentation  program  relates 
to,  supports,  and  leverages  the  activities  of  the  other  components  of  the  Joint  Vision  2010 
implementation  process.  USACOM  and  the  services  will  plan,  conduct  and  assess  concept-based 
experiments  to  determine  how  organization,  doctrine,  etc.  will  co-evolve  to  leverage  the  capacity 
of  emerging  technologies.  Experimentation  is  an  iterative  process  for  assessing  concept-based 
h5q)Otheses  to  identify  and  recommend  the  best  value-added  solution  for  change  to  doctrine, 
organization,  training,  material,  leadership  and  personnel  (DOTMLP)  required  to  achieve 
significant  advances  in  future  joint  operational  capabilities. 

The  analytic  community  must  posture  itself  to  address  the  analysis  of  the  results  of  these 
experiments.  This  special  meeting,  comprising  analysts  and  operational  experts,  will  concentrate 
on  assessing  and  improving  the  community’s  ability  to  plan,  conduct  and  analyze  the  results  of 
concept-based  experiments  laid  out  in  the  Joint  Experimentation  Campaign  Plan. 
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Goals  and  Objectives 


Goal 

The  goal  of  this  mini-symposium  and  workshop  is  to  contribute  to  planning,  conduct,  and 
exploitation  of  Joint  Experiments  by  leveraging  the  experience  and  expertise  of  the  analytical 
community. 

Objectives 

The  mini-s)Tnposium  will  inform  participants  about  the  nature  of  ongoing  military 
experimental  activities. 

The  workshop  will: 

Identify  ways  that  concept-based  experimentation  can  contribute  to  the  achievement  of  JV2010 
and  other  future  operational  concepts. 

1 .  Identify  appropriate  metrics,  methodologies  and  tools  for  concept-based  experiments. 

2.  Identify  the  critical  issues  in  planning,  conducting,  and  analyzing  the  results  of  experiments 
including  hypotheses  generation,  experimental  control  issues  (e.g.  man-in-the-loop,  free 
play),  measurement  and  methods  of  data  collection  and  analysis,  the  need  for  replicability, 
and  training  and  learning  curves. 

3.  Identify  ways  to  federate  and  utilize  existing  models  to  help  address  specific  hypotheses. 
Identify  opportunities  for  developing  strategic  joint  experimentation  plans  to  synchronize 
DoD  experimental  activities. 

4.  Identify  the  analytical  issues  associated  with  concept  development  and  prioritization. 

5.  Develop  pragmatic  guidelines  for  conducting  valid  experiments  whose  results  can  be  shared, 
while  supporting  the  needs  of  the  operational  community. 

6.  Evaluate  current  tools  and  models  to  assess  their  applicability  to  concept-based  experiments 
and  identify  requirements  for  future  analysis  tools  (e.g.  models,  simulations,  and  other 
analysis  tools  or  techniques). 

7.  Establish  the  practical  limits  of  modeling,  simulation,  and  analysis  as  applied  to  large  scale 
experiments. 

Approach 

This  will  be  a  three-phase  effort  consisting  of  pre-workshop  working  group,  a  mini-symposium, 
and  a  workshop.  The  pre-workshop  working  group  will  be  focused  upon  considering  the  nature 
and  role  of  experimentation  in  the  process  of  designing  and  developing  future  mission  capability 
packages.  The  results  of  this  working  group  effort  will  be  presented  as  a  point  of  departure  for 
the  workshop’s  deliberations.  The  mini-symposium  directly  preceding  the  workshop  will  consist 
of  a  number  of  presentations  that  address  ongoing  and  planned  experimental  activities  being 
undertaken  throughout  DoD.  The  workshop  will  be  organized  into  a  set  of  the  following  working 
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groups  that  will  examine  issues  related  to  Joint  Experimentation  from  a  number  of  different 
perspectives. 

1 .  Planning  Experiments:  Individually  and  as  part  of  a  campaign 

2.  Design  of  Experiments 

3 .  Preparation  &  Conduct  of  Experiments 

•  Site  and  Experimental  Preparation 

•  Data  Collection  and  Instrumentation 

•  Field  Test  Issues  and  Restrictions 

•  Experimented  Infrastructure  Requirements 

4.  Modeling/Simulation  &  War  Gaming  in  Support  of  Experimentation 

•  Balance  Constmctive  vs.  Live 

•  M  &  S  in  Experimental  Design  and  Concept  Development 

•  M  &  S  to  Exploit  Experimental  Results 

5.  Assessment  &  Utilization  of  Experimental  Results 

•  Analysis  and  Assessments  of  Concepts;  Integration  Across  Experiments 

•  "Validation"  of  Experiments 
Synthesis 

Tasking 

The  workshop  attendees  will  be  divided  into  working  groups  (15-18  members  each) 
to  examine  various  experimentation  issues  from  a  given  perspective.  Each  working 
group  will: 

1 .  Identify  the  issues  germane  to  their  area; 

2.  Consider  the  issues  in  light  of  the  goals  and  objectives  of  the  workshop; 

3.  Assess  our  ability  to  plan,  conduct,  and  analyze  within  the  context  of  the  working  group; 

4.  Identify  promising  approaches,  tools,  etc.; 

5.  Suggest  actions  that  need  to  be  taken  to  improve  our  ability  to  undertake  the  kind  of 
experimentation  program  needed. 
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List  of  Working  Group  Chairs  and  Co-Chairs 


MORS  Joint  Experimentation  Mini-Symposium  &  Workshop  Working  Groups  &  Chairs 


Tide 

Chair 

Co-Chair 

ACOM 

Co-Chair 

Synthesis  Group 
Representative 

Planning  Experiments 

Marion  Williams 

Col.  Steve  Myer 

Joe  Jennings 

Dave  Signori 

Design  of  Experiments 

Bob  Sheldon 

Daniel  Serfaty 

Capt  Johnson 

Dick  Hayes 

Preparation  &  Conduct  of 
Experiments 

Don  Eddington 

Richard  Kass 

COL  Geraci 

Ken  Jordan 

Modeling/Simulation  & 

War  Gaming  in  Support 
of  Experimentation 

David  Noble 

Bill  Stevens 

Maj  Ike 
Eichenberger 

Russ  Richards 

Assessment  &  Utilization 
of  Experimental  Results 

Jackie  Henningsen 

Priscilla  Glasow 

Linda  Weber 

Leland  Joe 

Synthesis  ' 

Stu  Starr 

Dave  Alberts 

Col  Gallinetti 

N/A 

Administration 

Name:  Joint  Experimentation 

Dates:  8-11  March  1999 

Location:  Armed  Forces  Staff  College 

Fee:  Mini-Symposium  Only:  Federal  Government  Employees:  $75;  All  others:  $150 

Mini-Symposium  and  Workshop:  Federal  Government:  $180;  All  others:  $360 
Attendance:  Mini-Symposium  —  Can  accommodate  up  to  500  people; 
Workshop  —  Limited  to  150  by  invitation  only 
Classification:  UNCLASSIFIED 
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Military  Operations  Research  Society 
Mini-SymposiumAV orkshop 
Joint  Experimentation 
Participants 
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MORS 

101  S.  Whiting  Street 

Alexandria  VA  22304 

OFF:  703-751-7290  FAX:  703-751-8171 

Email:morsvpa@aol.com 

Mr.J.  LAdkinson 

Marine  Corps  Warfighting  Lab  (EDOJSA) 

3255  Myers  Ave 
QuanticoVA  22134 

OFF:  703-784-5167  DSN:  278  FAX:  703-784-4597 
Email:adkinsonl@mcwl.quantico.mil 

DR  David  S  Alberts 
OASD(C3l) 

Room  3E172 

6000  Defense  Pentagon 

Washington  DC  20301-6000 

OFF:  703-695-7183  FAX:  ((703)  614-8060 

Email:David.Alberts@osd.pentagon.mil 

MR  William  R  Aldridge 
SETA  Corporation 
6862  Elm  Street 
McLean  VA  221 01 -3833 

OFF:  703-695-1763  DSN:  225-1763  FAX:  703-693-4601 
Email:aldridwr@js.pentagon.mil 

Dr  Robert  Steven  Alexander 
SAIC 

Suite  1010 
110  N  Glebe  Road 
Arlington  VA  22201 

OFF:  (703)-907-2547  FAX:  703-276-3055 
Email:ralexander@  sito.saic.com 


COL  Gary  Anderson 
Marine  Corps  Warfighting  Lab 
3255  Meyers  Ave 
Quantico  VA  22134 

OFF:  703-784-5096  DSN:  278  FAX:  703-784-4597 
Email:andersong@mcwl.quantico.usmc.mil 

MR  Gary  W  Arnett 
Synergy,  Inc. 

1763  Columbia  Road,  NW 
Washington,  DC  20309 
OFF:  (202)-232-6261  FAX:  (202)-232-8359 
Email:gamett@synergyinc.com 

Mr.  Edward  M  Ashley 
SAIC 

22  Enterprise  Pkwy 
Hampton  VA  23666 

OFF:  757-827-2620  FAX:  757-825-9104 
Email:edward.m.ashley@cpmx.saic.com 

Mr.  Grant  Ayers 
USACOM/J92 
PO  Box  51 302 
Ft  Monroe,  VA  23651-1302 

OFF:  757-726-6279  DSN:  680-6279  FAX:  757-726-6229 

MR  Dennis  R.  Baer 
Logicon 

2100  S  Washington  Blvd 
Arlington  VA  22204-5703 
OFF:  (703)-312-2149  FAX:  (703)-31 2-2780 
Email:  dbaer@logicon.com 

Mr.  John  Baird 
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Mr.  William  C  McDonald 
SPARTA,  Inc 
4901  Corporate  Drive 
Huntsville  AL  35805 

OFF:  256-837-5282  FAX:  256-830-0287 
Emaibmcdonaldb  @  huntsville.sparta.com 

Dr  Jimmie  G.  McEver  III 

RAND  Corporation 

1700  Main  St 

PO  Box  2138 

Santa  Monica  CA  90407 

OFF:  31 0-393-041 1  FAX:  310-451  -7073 

MAJ  Willie  JMcFadden  II 
US  Army  Student  Detachment 
Rm.  2041 
Bldg  5450 

Ft.  Jackson  SC  29270 

OFF:  757-877-6852  FAX:  757-877-6852 

Email:willemcfadden@mindspring.com 

MS  Lana  E  McGlynn 
ODUSA  (OR) 

Attn:  SAUS-OR,  RM  1E643 

102  Army  Pentagon 
Washington  DC  20310-0102 

OFF:  703-697-0367  DSN:  227-0367  FAX:  703-697-7748 
Email:mcglyla@hqda.army.mii 

Mr.  Larry  Mclnnis 
Atlantic  Intelligence  Command 
7941  Blandy  Road 
Norfolk  VA  23551 -2498 
OFF:  757-686-7626  FAX: 

Email:mcinnis@acom.mil 

Mr.  Douglas  A  McLiverty 
USACOM/J93 
5248  Rolleston  Dr 
Virginia  Beach  VA  23464 

OFF:  757-836-0915  DSN:  836  FAX:  757-836-0932 
Email:mcliverd@erols.com 


COL  Michael  K  Mehaffey 

US  Army  Training  &  Doctrine  Command 

Attn:  ATCD-B 

10  Whistler  Lane 

Fort  Monroe  VA  23651 -1046 

OFF:  757-727-2332  DSN:  680  FAX:  757-727-3445 

Emaihmehaffem  @  monroe.army.mi 

MR  Kenneth  J  Mellin 

COLSACORP 

STE505 

1235  Jefferson  Davis  Hwy 
Arlington  VA  22209 

OFF:  703-414-0940  FAX:  703-414-3683 
Emaihkmellin  ©erols.com 

Mr  Reed  Mihaloew 
MITRE  Corporation 
Suite  100 
234  S.  Fraley  Blvd 
Dumfries  VA  22026 

OFF:  703-441-7218  FAX:  703-441-1779 
Email:mihaloew@mitre.org 

COL  James  B  Mitcham 

HQTRADOC 

Attn:  ATAN-ZA 

Bldg  5G,  Room  303 

Ft  Monroe  VA  23651 -1048 

OFF:  757-728-5803  DSN:  680  FAX:  757-727-4394 

Emaihmitchamj  @  monroe 

Mr.  John  J  Moffatt 

AC2ISRC/AFEO 

STE216 

130  Andrews  St 

Langley  AFBVA  23665-1 993 

OFF:  757-825-8264  DSN:  574-9945  FAX:  757-825-9945 

Email:john.moffatt@  langley.af.mil 

CAPT  Steven  D  Monson 
OASD/C3I 

6000  Defense  Pentagon 
Washington  DC  20301-6000 
OFF:  703-695-0822  DSN:  225  FAX:  703-614-7247 
EmaihSteven.Monson  §  osd.pentagon.m  il 

MS  Kendra  E  Moore 
Alphatech,  Inc 
50  Mall  Road 

Burlington  MA  01 803-4537 

OFF:  781-273-3388  FAX:  781-273-9345 

Email:kendra.moore  §  alphatech.com 
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MAJ  Timothy  S  Moore 

AC2TSRC/AFEO 

STE216 

130  Andrews  St 

Langley  AFBVA  23665-1993 

OFF:  757-825-8234  DSN:  574-9945  FAX:  757-825-9945 

Eniail:timothy.moore@langley.at.mil 

Mr  Clarence  Todd  Morgan 

JTASC/JW940 

116  Lake  View  Pkwy 

Suffolk  VA  23435-2699 

OFF:  757  686-7497  FAX:  757  686-7985 

Email:morgan@jwfc.acom.mil 

Mr.  Charles  Morrisey 
Directorate  General  Strategic  Planning 
101  Col  By  Drive,  Pearkes  Bldg 
Ottawa,  Ontario  CANADA  KIA-OK2 
OFF:  613-996-6656  FAX:  613-992-5484 
Email:ae358 @  issc.debbs.ndhq  .dnd.ca 

MAJ  Robbie  L  Mosley 
OASD  (C3I) 

Crystal  Mall  3, 7th  Floor 
1431  Jefferson  Davis  Hwy 
Arlington  VA  22202 

OFF:  703-607-0263  DSN:  327  FAX:  703-607-0276 
Email:mosleyr@osd.pentagon.mil 

MR  Mark  W  Murray 

HQTRADOC 

Attn:  ATAN-A 

5G  North  Gate  Road 

Fort  Monroe  VA  23651-5000 

OFF:  757-728-5834  DSN:  680-5834  FAX:  757-727-4394 

Email:murraym@monroe.army.mil 

COL  Steven  H  Myer 
TRAC 

255  Sedgwick  Ave 
Attn:  ATRC 

Fort  Leavenworth  KS  66027-2345 

OFF:  913-684-6878  DSN:  552-6878  FAX:  913-684-6894 

Email:myers@trac.army.mil 

Mr.  Donald  W  Neal 
AFC2T1C/A7,  TRW  Team 
138  HartsonSt 
Hurlburt  Field  FL  32544 

OFF:  850-884-6837  DSN:  579  FAX:  850-884-1649 
Email:neal.donald@jfacc.hurlburt.af.mil 


Mr.  Richard  D  Newton 
USAF  C2TIC/A5 
138  Hartson  St 
Hurlburt  Field  FL  32544 

OFF:  850-884-6123  DSN:  579-6271  FAX:  850-884-6181 
Email:newton.rick@jfacc.hurlburt.af.mil 

MS  Cindy  Jahnke  Noble 

USA  TRAC 

Attn:  ATRC-SAA 

255  Sedgwick  Avenue 

Fort  Leavenworth  KS  66027 

OFF:  913-684-5511  DSN:  552-5511  FAX:  913-684-6894 

Email:noblec@trac.army.mil 

Dr  David  Noble 
Evidence  Based  Research 
Suite  250 

1595  Spring  Hill  Road 
Vienna  VA  22182 

OFF:  703-287-0312  FAX:  703-821-7742 
Emaiknoble  @  ebrinc.com 

LtCol  Robert  A  O'Neill 
ESC/DIS 

Chief  Modeling,  Simulation  &  Trng  Svc 
5  Eglin  Street 

Hanscom  AFB  MA  01 731-21 16 

OFF:  (617)-377-8486  DSN:  478-8486  FAX:  781  377-7469 

Emaihoneillr  @  hanscom.af.mil 

CAPTJD  Oliver 

J98-97  Assessments/Integration 

96  Fenwick  Road 

Ft.  Monroe  VA  23651-5000 

OFF:  757-726-6635  DSN:  680  FAX:  757-726-6181 

LTC  Maria  Chris  Pate 
DCSOPS 

400  Army  Pentagon 

Washington  DC  20310-0400 

OFF:  703-695-2459  DSN:  225-2459  FAX:  703-614-9044 

Email:patemac@hqda.army.mil 

MS  Anne  M  Patenaude 

SAIC 

Suite  290 

8301  Greensboro  Dr 
McLean  VA  22102 

OFF:  703-749-5109  FAX:  703-847-6406 
Email:anne.m.patenaude@cpmx.saic.com 
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MS  Lynda  Pedersen 

HQUSSOCOM 

Attn:  SOJ7-CS 

7701  Tampa  Point  Blvd 

MacDillAFBFL  33621 -5323 

OFF:  (813)-828-7705  DSN:  968-7705  FAX:  (813)-878-3880 
Email:pedersl@socom.mil 

MajGen  Timothy  Peppe 
USACOMJ9 

1562  Mitscher  Avenue,  Suite  200 
Norfolk  VA  23551 

OFF:  757-836-6472  FAX:  757-836-6478 
Email:peppe@acom.mil 


Mr.  Roger  Pratt 
SMA  Corporation 
901  Follin  Lane 
Vienna  VA  221 80 

OFF:  703-242-4681  FAX:  703-255-3667 
Email:roger.pratt@smawins.com 

COL  Roderick  D  Reay 
AFOTEC/OL-VA 
Suite  200  (J96) 

1562  Mitscher  Avenue 
Norfolk  VA  23551 

OFF:  757-836-0721  DSN:  836-0721  FAX: 
Emaihreayr  @  afotec.af.mil 


Mr.  Robert  W  Peterman 
Air  Force  Experimentation  Office 
2101  Executive  Drive 
STE  2D,  Tower  box  87 
Hampton  VA  23666 

OFF:  757-825-8234  DSN:  574-9945  FAX:  757-825-6214 
Email:robert.peterman@  langley.af.mil 


LCDR  Michael  L  Reynolds 
Atlantic  Intelligence  Command 
STE  100 

7941  Blandy  Road 
Norfolk  VA  23551-2498 
OFF:  757-836-7079  FAX:  757-836-7167 
Emaikreynolds  @  aic.ocom.mil 


DR  Steven  E  Pilnick 

EDO  Technology  Services  and  Analysis 

West  Bldg 

7600  Leesburg  Pike 

Falls  Church  VA  22043-2004 

OFF:  (703)-714-1874  FAX:  (703)-714-1801 

Email:spilnick@edo-tsa.com 

Mr.  Alan  W  Plotnik 

Raytheon  Systems  Company 

STE  812 

2  Eaton  Street 

Hampton  VA  23669-4054 

OFF:  757-224-061 2  FAX:  757-722-6746 

Email:awplotnik@west.raytheon.com 

Dr  Isaac  R  Porche  III 
RAND 

1700  Main  Street,  MS  23a 
PO  Box  2138 

Santa  Monica  CA  90407-2138 

OFF:  31 0-393-041 1  FAX:  31 0-451  -7073 

Email:porche@rand.org 

Mr  Robert  J  Powers 
SAIC 

4501  Daly  Drive,  Suite  400 
Chantilly  VA  20151-3707 
OFF:  703-808-6547  FAX: 
Email:Robert.J.Powers@cpmx.saic.com 


DR  Roy  E  Rice 

Teledyne  Brown  Engineering 

PO  Box  070007 

Huntsville  AL  35807-7007 

OFF:  256-726-2038  FAX:  256-726-2241 

Email:roy_rice@pobox.tbe.com 

DR  F.  Russell  Richards 

MITRE 

STE  400 

7941  Blandy  Road 

Norfolk  VA  23551-2498 

OFF:  757-836-221 1  FAX:  757-836-6478 

Email:rrichard@mitre.org 

Mr  Jon  A  Roth 
The  MITRE  Corporation 
Suite  100 
234  Fraley  Blvd 
Dumfries  VA  22026 

OFF:  703-441-1775  FAX:  703-441-1779 
Email:jroth@  mitre.org 

Mr.  Connie  W  Sawyer  Jr 
US  GAO 

5029  Corporate  Woods  Dr 
Virginia  Beach  VA  23462 
OFF:  757-552-8140  FAX:  757-552-8197 
EmaiLsawyerc.noro  §  gao.gov 
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Mr  Randall  Schmidt 
AC2ISRC/AFE0 
2101  Executive  Drive 
Hampton  VA  23666 

OFF:  757-765-9945  DSN:  574-9945  FAX:  757-765-9945 
Email:randall.schmidt@langley.af.mil 

Mr.  William  H  Schopfel 
The  MITRE  Corporation 
PO  Box  8068 

Camp  Lejeune  NC  28547-8068 
OFF:  910-451-5869  FAX:  910-451-9422 
Emaihschopfel  @  mitre.org 

MR  Albert  A  Sciarretta 
Quantum  Research  International 
Suite  700 
9302  Lee  Hwy 
Fairfax  VA  22031 

OFF:  703-218-2445  FAX:  (703)  383-4892 
Email:asciarretta@quantum-intl.com 

DR  Gregory  Searle 

Embassy  of  Australia 

1601  Massachusetts  Ave,  NW 

Washington  DC  20036 

OFF:  (202)-797-3378  FAX:  (202)-797-1838 

Email:condsa  @  clark.net 

Mr  Daniel  Serfaty 
Aptima,  Inc 
STE  3050 

600  West  Cummings  Park 
Burlington  MA01801 
OFF:  781-935-3966  FAX:  781-935-4385 
Email:serfaty@aptima.com 

LtCoI  Joseph  E  Sharpless 
DIA 

Fenwick  Road,  Bldg  96 

Ft.  Monroe  VA  23651 -5000 

OFF:  757-726-6426  DSN:  680  FAX:  757-726-6433 

Email:sharplej  ©jwfc.js.mil 

DR  Robert  S  Sheldon 
S31 

Suite  500 

1700  Diagonal  Road 

Alexandria  VA22314 

OFF:  703-684-4529  FAX:  703-684-8272 

Email:bs@s3i.com 


DR  David  T  Signori  Jr 

RAND  Corporation 

1 333  H  Street,  NW 

Washington  DC  20005-4707 

OFF:  202-296-5000  DSN:  426-2235  FAX:  202-296-7960 

Emaiksignori  @  rand.org 

Mr.  Richard  A  Simpson  (UK) 

Allied  Command  Atlantic  (SACLANT) 

STE  100 

7857  Blandy  Road 
Norfolk  VA  23551-2490 

OFF:  757-445-3885  DSN:  565  FAX:  757-445-3242 
Emaiksimpson  @  saclant.nato.int 

DR  Cyrus  J.  Staniec 
Logicon 

2100  Washington  Blvd 

Arlington  VA  22204-5706 

OFF:  (703)-31 2-3500  FAX:  (703)-31 2-2780 

Emaikcstaniec@logicon.com 

DR  Stuart  H  Starr  FS 
The  MITRE  Corporation 
1 820  Dolley  Madison  Blvd  #MS  W557 
McLean  VA  22102 

OFF:  (703)-883-5494  FAX:  (703)-883-1373 
Emaikstarr  @  mitre.org 

Mr.  Michael  D  Starry 
IDA 

1 23  Sarazen  Court 

Newport  News  VA  23602 

OFF:  757-836-0696  FAX:  757-836-0935 

Emaikmstarry@ida.org 

MR  Malcolm  P  Taylor 

Pacific  Sierra  Research  Corporation 

STE  700 

1400  Key  Blvd 

Arlington  VA  22209 

OFF:  703-516-6202  FAX:  703-524-2420 
Emaikmtaylor  @  psrw.com 

MR  Donald  W  Theune 
Virtual  Technology  Corp 
Suite  114 

5400  Shawnee  Road 

Alexandria  VA  22312-2300 

OFF:  (703)-658-7943  FAX:  (703)-658-7057 

Emaikdtheune@virlc.com 
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MR  Clayton  J  Thomas  FS 

AFSAA/SAN 

1570  Air  Force  Pentagon 

Washington  DC  20330-1 570 

OFF:  703-588-6944  DSN:  425-6944  FAX:  703-588-8779 

Email:CIayton.Thomas@pentagon.af.mil 

Col  Lynne  C  Thompson 
HQAF/XOCP 
Alexandria  VA  22312 
OFF:  703-588-5010 

LTC  Fletcher  B  Thornton 
The  Joint  Staff-J8 
LLWAD 

8000  Pentagon,  Room  1D940 

Washington  DC  20318-8000 

OFF:  703-695-4657  DSN:  225-4657  FAX;  703-695-8031 

Email:thorntfb@js.pentagon.mil 

Mr  James  D  Touhill 
Quantum  Research  International 
Crystal  Square  4,  Surte  1001 
1745  Jefferson  Davis  Hwy 
Arlington  VA  22202 

OFF:  757-727-3551  DSN:  680-3551  FAX:  757-727-2947 
Emaihtouhillj  @  monroe.army.mil 

Mr.  Joseph  Trujillo 
TRADOC  Analysis  Center 
Attn;  ATRC-WAA 

White  Sands  Missile  Range  NM  88002-5502 
OFF:  505-678-1558  DSN:  258  FAX:  505-678-1450 
Email:trujillj  ©trac.wsmr.army.mil 

LTC  David  W  Tyner 

Joint  National  Test  Facility  (BMDO) 

730  Irwin  Ave 

Falcon  AFB,  CO  80912-7300 

OFF;  719-567-8326  FAX:  719-567-9219 

MR  Stephen  C  Upton 

Los  Alamos  National  Lab 

TSA5,  MSF602 

PO  Box  1663 

Los  Alamos  NM  87544 

OFF:  (505)-667-9435  FAX:  (505)-665-2017 

Email;upton@lanl.gov 


MS  Gwendolyn  D  Vantree 
HQ  TRADOC,  DCSSA 
ATTN:ATAN-SM 
Bldg  5G,  3rd  Floor 
Ft  Monroe  VA  23651-1048 

OFF:  757-728-5833  DSN:  680-5833  FAX:  757-751-4394 
Email:vantree@monroe.anny.mil 

Mr.  Charles  D  Vollmer 
Booz  Allen  &  Hamilton 
OFF:  703-902-5541  FAX:  703-902-3374 

MR  Charles  E  Walters 
OSD  JTE&E  Program  Office 
STE800 

2001  N  Beauregard  St 

Alexandria  VA  22311 

OFF;  703-578-6122  FAX;  703-578-6172 

Ms.  Delores  Washburn 

Space  and  Naval  Warfare  System  Command 

PMW132 

4301  Pacific  Hwy 

San  Diego  CA  92110-3127 

OFF:  619-537-0194  FAX;  619-537-0155 

Email:delores  @  spawar.navy.mil 

Ms  Linda  L.  Weber 
USACOMJ97/MITRE 
Atlantic  Intelligence  Command 
7941  Blandy  Road,  Suite  400 
Norfolk  VA  23551 -2498 

OFF:  757-726-6161  DSN:  680-6161  FAX:  757-726-6181 
Email:weberl@  jwfc.js.mil 

RADM  Gary  F  Wheatley  (USN-Ret) 

Evidence  Based  Research,  Inc 
STE250 

1595  Spring  HilIRd 
Vienna  VA  221 82-2228 
OFF:  703-893-6800  FAX;  703-821  -7742 
Emaihwheatley  @  ebrinc.com 

MR  Richard  I  Wiles 
Military  Operations  Research  Society 
101  S  Whiting  St  #202 
Alexandria  VA  22304 

OFF:  (703)-751-7290  FAX:  (703)-751-8171 
Emaihmorsone  @  aol.com 
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CAPT  Richard  B  Williams 
SPAWARSYSCOM 
PMW132  (C60/317) 

4301  Pacific  Hwy 

San  Diego  CA  921 10-31 27 

OFF:  619-537-0197  DSN:  577  FAX:  619-537-0153 

Email:wiilrb  @  spawar.navy.mil 

MS  Lauran  Winter 

HQTRADOC 

ODCSSA 

ATTN:  ATAN-SM 

Fort  Monroe  VA  23651-5143 

OFF:  DSN:  680-5953  FAX:  717-245-3603 

Email:winterl@awc.carlisle.army.mil 

Mr.  Dale  R  Winzer 

AC2ISRC/AFEO 

STE216 

130  Andrews  St 

Langley  AFBVA  23665-1 993 

OFF:  757-825-8234  DSN:  574-9945  FAX:  757-825-9945 

Email:dale.winzer@langley.af.mil 

LTC  Mark  S  Woempner 
TRAC-WSMR 
Bldg  1400 

White  Sands  Missile  Range  NM  88002 

OFF:  505-678-2956  DSN:  258  FAX:  505-678-1450 

Email:woempnem@trac.wsmr.army.mil 

DR  D.  Robert  Worley 
IDA/CSED 

1 801  N  Beauregard  Street 
Alexandria  VA  2231 1 

OFF:  (703)-845-6631  FAX:  (703)-845-6848 
Email:rworleyda.org 

MR  Michael  E  Wright 
The  MITRE  Corporation 
MS  1302 

202  Burlington  Road 
Bedford  MA  01730 

OFF:  781-377-8505  FAX:  781  377-7469 
Email:mewrighf@mitre.org 

MS  Susan  J  Wright 

Institute  for  Defense  Analyses 

1801  N.  Beauregard  Street 

Alexandria  VA  2231 1-1772 

OFF:  703-845-6628  FAX:  703-845-6809 

Email:sjwright@ida.org 


Maj  Chris  Yunker 
Office  of  Naval  Research 
800  N  Quincy  Street 
Arlington  VA  2221 7-5660 

OFF:  703-696-7020  DSN:  426-7020  FAX:  703-696-4162 
Email:yunderc@onr.nary.mil 

Ms.  Anita  A  Zabek 
The  MITRE  Corporation 
1820  Dolley  Madison  Blvd 
McLean  VA  221 02 

OFF:  703-883-1389  FAX:  703-883-6435 
Email:aradams@mitre.org 

LtCol  Frank  A  Zawada 

AFRL/VSSW 

29  Randolph  Road 

Hanscom  AFB  MA  01731-3010 

OFF:  781)-377-8004  DSN:  478-8004  FAX:  781)-478-5640 

Email:zawada@plh.af.mil 

Mr.  Steven  R.  Zeswitz 
The  MITRE  Corporation 
PO  Box  8068 
Camp  Lejeune  NC  28547 

OFF:  910-451-5869  DSN:  751-5869  FAX:  910-451-9422 
Email:zeswitz@  mitre.org 
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